alexmandClaude
a09d158e16
Keys (spec/02 §2.4), the rule paperstack settled on:
- A work key is a Scholar-style citation key, a hyphen and the first four
Crockford base32 characters of SHA-256("scienting-key/1:" + identity),
where the identity is the first of doi, arxiv, isbn, pmid, pmcid and
openalex in same_as, else the folded title, year and family name. A
collision lengthens the later key by two characters; a key is minted
once and never recomputed. A document key adds .si, .v<seq> or .<type>
for versions other than the version of record.
- Every fixture key is now minted (hale2026harmonic-damb,
okafor2025solvent-acth, writer2027fixed-ynr7.v3, tiny2026tiny-j4qm), so
every node and record ID was re-minted and the docs rewritten to match.
The validator checks the document key against the work key and version
(doc.key.work) and the hash part against the identifier (doc.key.suffix).
vectors/keys.jsonl pins the cases paperstack's tests pin.
Review:
- Spot-check seeds are ECVRF-EDWARDS25519-SHA512-TAI proofs (RFC 9381),
implemented in tools/oracle/vrf.py in plain Python and reproduced
against the RFC's Example 16; the sample ranks by the VRF output. The
earlier Ed25519 seed could be ground by a dishonest checker.
- RFC 9162 inclusion and consistency proofs (crypto.inclusion_path,
verify_inclusion, consistency_proof, verify_consistency) with
scienting-proof/1, vectors/proofs.jsonl and two fixture records.
- document.json carries `history`, the root over the corrections, with
the empty-tree hash for none (spec/11 §11.5, rule doc.history).
- A provisional transparency log (spec/23): log entries, signed-note
checkpoints with key hashes as transparency logs share them, the rule
that a logged revocation beats a self-asserted time, and vectors.
- Reserved signature schemes cose and ml-dsa-65 (spec/08 §8.4).
Identifiers and text:
- urn:scienting: becomes tag:scient.ing,2026: (RFC 4151), since the URN
namespace was never registered.
- Unicode 16.0 is pinned for normalization and categories (spec/01 §1.4);
the checker refuses another version.
Checking:
- check.py: keys, proofs, VRF and checkpoint vectors; the proof and
checkpoint records; a check that every ID and hash the prose quotes
exists in the fixtures or vectors; three more broken packages
(history-wrong, key-suffix, key-work).
- stress.py: key, VRF, proof and checkpoint properties.
- Docs: a keys section in the concepts and FAQ, rationale entries for
keys, VRF, proofs, tag URIs and the Unicode pin, a "Looking ahead"
section, media-type registration and OCI distribution notes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
186 lines
15 KiB
Markdown
186 lines
15 KiB
Markdown
# Walkthrough: one paper, start to finish
|
||
|
||
This guide follows the example paper `hale2026harmonic-damb` from a flawed first reading to a corrected, reviewed and signed document.
|
||
The paper is synthetic, written for the fixtures.
|
||
Its problems are modeled on the failures that the specification's rules address.
|
||
|
||
Three readers can use the guide, and each gets something different.
|
||
A newcomer knows what a paper and a citation are, not hashes or JSON, so every term is defined where it first appears.
|
||
An implementer who will write a reader, writer, converter or checker gets the chapter and test vector for each step.
|
||
An expert gets the status: [paperspec](https://git.scient.ing/infra/paperspec) is at version 0.1.0, and each format is `candidate` or `provisional` in [`registry/formats.json`](../registry/formats.json) (19 candidate, 13 provisional).
|
||
The `scienting-` prefix on format names, as in `scienting-doc/1`, is the formats' namespace.
|
||
It has nothing to do with the repository's name.
|
||
|
||
Open [fixtures/packages/hale2026harmonic-damb/](../fixtures/packages/hale2026harmonic-damb/) beside this page, because every ID below is in it.
|
||
A package is a folder of files, mostly JSON Lines (`.jsonl`: one JSON record per line).
|
||
The hand-written source of the first reading is [fixtures/src/hale2026harmonic.json](../fixtures/src/hale2026harmonic.json); no converter ran on it, and [`tools/build.py`](../tools/build.py) expands it into the package.
|
||
Each step says what a converter, a person or a checker does.
|
||
|
||
## 1. A PDF arrives
|
||
|
||
The team downloads the publisher's PDF.
|
||
The file becomes a witness, identified by its SHA-256 (a 64-digit hash that changes if any byte changes), with the short ID `qptsanv` ([spec/02](../spec/02-model.md)).
|
||
A person records its rights, meaning what the team may do with the file ([spec/14](../spec/14-rights.md)).
|
||
The library's subscription allows reading, text mining, and AI processing on the team's own machines for internal use, until August 31, 2027.
|
||
Whether it allows processing in the team's cloud is unknown.
|
||
|
||
The publisher stamped the download in the margin with the institution and the time, so two downloads rarely share a SHA-256.
|
||
A page fingerprint avoids the problem of downloads that differ in their stamps.
|
||
It is a hash of a page's text, size and images after the stamp lines are removed ([spec/07](../spec/07-geometry.md)), and the rules `downloaded-via` and `sharing-guidelines` fired on both pages (see [`pages.jsonl`](../fixtures/packages/hale2026harmonic-damb/pages.jsonl)).
|
||
Page 101's fingerprint is `6272e6f3…` in this copy, stamped October 11, 2026, and in a second download from another institution, stamped March 2, 2027 ([vectors/fingerprints.jsonl](../vectors/fingerprints.jsonl)).
|
||
|
||
## 2. The first reading, with its mistakes
|
||
|
||
A converter reads the PDF into sections, paragraphs, sentences, a figure, a table, an equation, a footnote and references.
|
||
Each of these is a node, and each node gets a permanent ID: seven characters minted once from the document key, the node's kind, a hash of its text and an occurrence count, then frozen ([spec/04](../spec/04-identity.md)).
|
||
An ID encodes no page, order or kind, so it keeps naming the node after the text is corrected.
|
||
|
||
Like every reading, this one has mistakes:
|
||
|
||
| What the reading did | What the page says |
|
||
| --- | --- |
|
||
| A cell read `3l0.0`. | `310.0`, with the digit 1. |
|
||
| A sentence read `Values below 10-3 kcal mol⁻¹…`. | A superscript: 10⁻³. |
|
||
| Two sentences ran together: "…treated as zero. Table 1 lists…". | Two sentences. |
|
||
| One sentence was cut in two at a line break: "…and the energies were" and "fitted by least squares." | One sentence. |
|
||
| A paragraph was split at a column break. | One paragraph. |
|
||
| The running head "J. Example Chem. 2026, 14, 101–109" became a paragraph. | A running head, not text. |
|
||
| The equation's number read `(l)` and the figure's `Figure l`. | `(1)` and `Figure 1`. |
|
||
| The CT–HC row merged two cells into "338.7 1.0900". | Two cells. |
|
||
| The OW–HW row was missing. | It is there. |
|
||
| "Set B" spanned one column. | It spans two. |
|
||
|
||
The converter does not guess.
|
||
Each doubtful place becomes a suspect, with a crop (an image cut from the page) for a person to answer ([spec/05 §5.7](../spec/05-hashes.md)).
|
||
Unresolved suspects travel in `suspects.jsonl`, which this package does not need.
|
||
Nothing changes until a correction passes the gate ([spec/11](../spec/11-corrections.md)).
|
||
|
||
## 3. Twelve corrections
|
||
|
||
Each fix is appended to [`corrections.jsonl`](../fixtures/packages/hale2026harmonic-damb/corrections.jsonl), numbered, with its evidence (a crop, or a person who read the page), its author (`ana` throughout) and its reason.
|
||
|
||
| # | Operation | On | What it did |
|
||
| --- | --- | --- | --- |
|
||
| 1 | `replace` | cell `c20cna6` | `3l0.0` to `310.0`. A digit changed, so the gate required a crop, and it has one. |
|
||
| 2 | `replace` | sentence `ec5jc93` | `10-3` to the runs `10` and superscript `−3`. |
|
||
| 3 | `split` | sentence `ec5jc93` | Split before "Table 1"; the first part keeps `ec5jc93`, the second is new: `a4jrz2t`. |
|
||
| 4 | `merge` | sentence `xjns57z` | Joined with `ayea71r`, which is retired with `xjns57z` as its successor. |
|
||
| 5 | `merge` | paragraph `gwzbj3s` | Joined with `ncx2v4f`, retired the same way. |
|
||
| 6 | `delete` | paragraph `v8e4z6z` | The running head is gone; its IDs are retired with no successor. |
|
||
| 7 | `relabel` | equation `tkxsmts` | `(l)` to `(1)`. |
|
||
| 8 | `delete` | row `rncchma` | The garbled CT–HC row and its four cells. |
|
||
| 9 | `insert` | row `kx2kz2y` | The CT–HC row, read again: five new cells. |
|
||
| 10 | `insert` | row `ke0jxpn` | The missing OW–HW row. |
|
||
| 11 | `set` | cell `xyy7x6c` | Set B's `colspan` from 1 to 2. |
|
||
| 12 | `relabel` | figure `v7cx6qs` | `Figure l` to `Figure 1`. |
|
||
|
||
The document is now at revision 12.
|
||
No uncorrected cell, sentence or row changed its ID or its hashes.
|
||
Only corrected nodes and their ancestors got new `node_hash` values; the sentence `rr20tev` is one of the ancestors, because it holds the relabeled equation.
|
||
Inserting a row adds a row node.
|
||
The table's own hash (`own_hash`) covers only its label and column definitions, so every other cell, and every value already taken from them, is untouched.
|
||
|
||
## 4. IDs that never move
|
||
|
||
The sentence about water is still `hca6dc5` after twelve corrections, the corrected cell is still `c20cna6`, and the sentence that lost its superscript is still `ec5jc93`.
|
||
The retired IDs are recorded in [`ids.jsonl`](../fixtures/packages/hale2026harmonic-damb/ids.jsonl), so anything that pointed at them can follow the trail.
|
||
Legacy aliases are recorded there too: `b-p01-text-003`, the ID an older tool gave the water sentence, still leads to `hca6dc5`.
|
||
|
||
A fresh conversion of the same input mints the same IDs.
|
||
A reading by a different converter is aligned with this document sentence by sentence ([spec/05 §5.6](../spec/05-hashes.md), pinned by [vectors/alignment.jsonl](../vectors/alignment.jsonl)).
|
||
Sentences the alignment pairs keep their IDs, and every difference becomes a suspect rather than a silent change.
|
||
Sentences it cannot pair are recorded as dropped or added.
|
||
|
||
## 5. What happened to the notes
|
||
|
||
An anchor is the stored pointer behind a note or citation.
|
||
It holds a node's ID, the node's hashes when the anchor was made, the revision, and, for part of a sentence, the quoted words with context ([spec/06](../spec/06-anchors.md)).
|
||
The team had already made seventeen anchors pointing into the first reading at revision 0, before any correction.
|
||
|
||
Resolving an anchor gives one of five results:
|
||
|
||
- `current`: nothing changed.
|
||
- `changed-below`: the node is unchanged, and something under it changed.
|
||
- `moved`: the node was merged into a successor that still holds the quote.
|
||
- `reattached`: the node changed, and the quote or the cell's headers now find exactly one place.
|
||
- `stale`: nothing is found, so a person must look.
|
||
|
||
A `changed-below` result carries a drift, the share of the leaves under the node, counting retired ones, that corrections touched.
|
||
Above a quarter the drift is `high`.
|
||
The table shows how each anchor resolves at revision 12 ([vectors/resolve.jsonl](../vectors/resolve.jsonl) records them all).
|
||
|
||
| Anchor on | Result | Why |
|
||
| --- | --- | --- |
|
||
| "H$_2$O" in the water sentence | `current` | Nothing in that sentence changed. |
|
||
| The cell `1.5260` | `current` | Untouched, though its table changed around it. |
|
||
| The water sentence, through the alias `b-p01-text-003` | `current` | The alias leads to the same node. |
|
||
| The cell that read `3l0.0` | `stale` | Its text changed and "3l0.0" is no longer in it; a person must look again. |
|
||
| "treated as zero" | `reattached` | The sentence changed twice, and the quote occurs in it exactly once, so the anchor is re-pinned. |
|
||
| "fitted constants" | `stale` | After the split, those words are in the new sentence `a4jrz2t`, not in `ec5jc93`. |
|
||
| "least squares", in the half-sentence `ayea71r` | `moved` | That ID was merged into `xjns57z`, which still holds the quote. |
|
||
| The paragraph `ncx2v4f` | `moved` | Merged into `gwzbj3s`, which holds all of its sentences. |
|
||
| The running head | `stale` | Deleted, with no successor. |
|
||
| The equation | `reattached` | Only its number changed. |
|
||
| The figure | `reattached` | Only its label changed, by a `relabel`. |
|
||
| The CT–HC cell `1.090` in the garbled row | `reattached` | Its row was deleted, and exactly one cell in Table 1 has the same row and column headers and the same text: the new `b5gd0h1`. |
|
||
| Section 2 | `changed-below`, drift high | The section itself is unchanged, and 53% of what it holds or held was touched. |
|
||
| Table 1 | `changed-below`, drift high | Rows were replaced and added. |
|
||
| The table body | `changed-below`, drift high | 75% of the cells it holds or held were touched. |
|
||
| The paragraph whose broken sentence was rejoined | `changed-below`, drift high | Two of the three sentences it has held were touched. |
|
||
| The CT–CT row | `changed-below`, drift low | One of its five cells was corrected. |
|
||
|
||
A reader shows each note by its result.
|
||
A checker of exact words accepts only `current`, `moved` and `reattached`.
|
||
|
||
The [W3C Web Annotation Data Model](https://www.w3.org/TR/annotation-model/) is the general W3C Recommendation for annotating web resources. paperspec's annotation layers build on it, with a custom `AnchorSelector` followed by a standard `TextQuoteSelector` as a fallback.
|
||
It has no permanent IDs for sentences inside the source, no hashes of the target text, and no results such as `moved` or drift.
|
||
See [comparison.md](comparison.md).
|
||
|
||
## 6. Review and signatures
|
||
|
||
A receipt is a signed record that a person compared a set of nodes with the page.
|
||
It lifts a node to F3, the top fidelity level (F0 is one reading; F1, every doubtful place answered; F2, an independent witness agrees) ([spec/08](../spec/08-fidelity.md)).
|
||
One signature covers a Merkle tree, a hash over everything reviewed in a sitting, built as in RFC 6962 (a Request for Comments standard; replaced by [RFC 9162](https://www.rfc-editor.org/rfc/rfc9162), which defines the same tree).
|
||
The receipts are in [`receipts.jsonl`](../fixtures/packages/hale2026harmonic-damb/receipts.jsonl), and the keys that signed them in [`roster.json`](../fixtures/packages/hale2026harmonic-damb/roster.json).
|
||
|
||
- **Before the corrections**, `ana` had signed receipt `npc15pn6675z5` for three cells at revision 0.
|
||
Two of them never changed, so the receipt still counts for them.
|
||
For the corrected cell it no longer counts, because the cell's hash moved on.
|
||
- After the corrections, `ana` signs `y2zw8ctg5s8dp` at F3 for 23 nodes: the cells of Table 1's three data rows (15), its four column headers, the equation, and three sentences (`ec5jc93`, `hca6dc5` and `rr20tev`, the one holding the equation).
|
||
`ben` spot-checks it with receipt `mg18668nzxvap`.
|
||
His signature, made with his separate spot-check key `ben-spot`, picks 10 nodes nobody could have chosen, and he finds no errors.
|
||
- `ana` signs `mvjmb94nz7rdh` for two sentences of section 1, including the new `a4jrz2t`.
|
||
- `cara` signs `q465jpwz8afhe` for three sentences on page 2, then revokes it herself (`pcbhtpwvysgq6`) because she had compared them with the crops of another paper.
|
||
Those sentences fall back to their earlier levels, which is why `xjns57z` is still F0.
|
||
- `ana` signs `gx2s84z9jf9xv` for the figure's three caption sentences. `ben`'s spot-check `yt70fcc3qm0xb` finds a missing period in `caehhp6`, which suspends the receipt until she reviews the sitting again and signs a new one.
|
||
|
||
The document's report, in [`document.json`](../fixtures/packages/hale2026harmonic-damb/document.json), counts the result over 45 leaves: 27 at F3, 13 at F2, 4 at F1 and 1 at F0.
|
||
The 27 are the 23 nodes of `y2zw8ctg5s8dp`, the 2 of `mvjmb94nz7rdh`, and 2 of the 3 caption sentences.
|
||
Those two still count because Ben's spot-check is itself a valid receipt, and he compared them with the page and found them right.
|
||
The third, `caehhp6`, stays at F2: a spot-check never raises a node it found in error ([spec/08 §8.2](../spec/08-fidelity.md#82-levels)).
|
||
|
||
## 7. Data out of the table
|
||
|
||
Table 1 becomes a dataset, `ds-hale-bond-params`, inside the package ([spec/15](../spec/15-datasets.md)).
|
||
It is a [Frictionless Data Package](https://datapackage.org/standard/data-package/) with units, uncertainty and an origin added, and the format is `provisional`.
|
||
Its CSV keeps every digit as printed (`1.526` and `1.5260` stay different), and every value has an anchor to the cell that prints it, so a claim that rests on a value can show the cell.
|
||
The descriptor types the fields as numbers; how a generic Data Package tool treats `1.5260` was not verified.
|
||
All twelve value anchors resolve as `current`.
|
||
The reference checker, [`tools/check.py`](../tools/check.py), confirms each CSV value equals its cell's text.
|
||
|
||
## 8. Sharing without the text
|
||
|
||
The team wants to move its notes to another of its own machines without copying the paper there.
|
||
[`hale2026harmonic-damb.checks-only`](../fixtures/packages/hale2026harmonic-damb.checks-only/) is the same document with every leaf's text withheld and every hash replaced by an HMAC under the team's key ([spec/12 §12.5](../spec/12-package.md), [spec/05 §5.8](../spec/05-hashes.md)).
|
||
An HMAC is a hash mixed with a secret key.
|
||
Anyone can recover a plain hash of a short text such as "310.0" by hashing guesses; a keyed hash resists that.
|
||
Corrections, receipts, the roster and the dataset are left out too.
|
||
IDs, labels, pages and boxes survive, so annotations still land in the right place.
|
||
Only someone holding the key and the text can check the values.
|
||
The team key suits an export between the team's own machines.
|
||
An export to a collaborator outside the team would use a key derived from the pages of the paper instead, so that only someone holding the paper can check it.
|
||
|
||
## 9. Recompute, print and query the package
|
||
|
||
[The examples](examples.md) show how to recompute one of these hashes with `sha256sum`, print the outline with a few lines of Python, and load the whole package into SQLite to query it.
|