263 lines
15 KiB
Markdown
263 lines
15 KiB
Markdown
# Concepts
|
||
|
||
Each section introduces one idea, shows it in the example paper `hale2026harmonic` (a synthetic chemistry paper in [fixtures/packages/](../fixtures/packages/)), and links to the chapter that defines it exactly.
|
||
|
||
## Works, versions, witnesses and documents
|
||
|
||
Four things are easy to confuse, so the formats keep them apart.
|
||
|
||
| Object | In the example |
|
||
| --- | --- |
|
||
| A **work** is the paper across all its versions. | Hale and Osei's paper on harmonic bond parameters, `doi:10.5555/example.hale.2026`. |
|
||
| A **version** is one expression of it: a preprint, the version of record, an erratum, the Supporting Information. | The version of record. |
|
||
| A **witness** is one file of a version, identified by its SHA-256. | The publisher's PDF, witness `qptsanv`. |
|
||
| A **document** is the text established for one version, from one or more witnesses. | `hale2026harmonic`. |
|
||
|
||
A document's key is its work's key, with a suffix for anything but the version of record: `hale2026harmonic.si` would be its Supporting Information.
|
||
See [chapter 2](../spec/02-model.md).
|
||
|
||
## A document is a tree of nodes
|
||
|
||
Every part of the text is a node: the document, its sections, headings, paragraphs and sentences, figures and their panels and captions, tables with their rows and cells, equations, footnotes and references.
|
||
The example has 75 nodes; its outline is:
|
||
|
||
```text
|
||
bzqvbhd 1 Introduction
|
||
qn08tj9 equation (1)
|
||
f0tj6ca 2 Methods
|
||
ajfvq4s figure Figure 1
|
||
gm5z4rx table Table 1
|
||
```
|
||
|
||
`nodes.jsonl` holds one node per line, in reading order, each with its kind, parent, position, content and hashes.
|
||
A sentence's content is a list of **runs**: plain text, styled text, inline math, citations, cross-references, footnote marks, links, and display equations the sentence runs through.
|
||
This is one table cell, shortened:
|
||
|
||
```json
|
||
{"id":"nkm3xn5","kind":"cell","parent":"zhe4g0k","ord":1,
|
||
"attrs":{"column":"…","role":"data","rowspan":1,"colspan":1},
|
||
"content":[{"kind":"text","text":"310.0"}],
|
||
"fidelity":"F1","text_hash":"…","match_hash":"…","align_key":"…","own_hash":"…","node_hash":"21f13b…"}
|
||
```
|
||
|
||
See [chapter 3](../spec/03-nodes.md).
|
||
|
||
## IDs are names, not positions
|
||
|
||
`nkm3xn5` says nothing about where the cell is.
|
||
That is deliberate: an ID that encoded a page or an ordinal would change when a better converter ordered blocks differently.
|
||
IDs are minted once from the document key, the node's kind, its text and an occurrence count, so a fresh conversion of the same input gives the same IDs, and then they never change.
|
||
|
||
When two nodes merge, one ID is retired and points at its successor; when an artifact is deleted, its ID is retired with no successor.
|
||
Legacy IDs, such as the block IDs older tools used, live on as aliases: in the example, `b-p01-text-003` still leads to sentence `f929eq9`.
|
||
See [chapter 4](../spec/04-identity.md).
|
||
|
||
## Addresses
|
||
|
||
An address names a node, and optionally pins it to a state and narrows it to a span:
|
||
|
||
| Address | Names |
|
||
| --- | --- |
|
||
| `hale2026harmonic#f929eq9` | The sentence about water, whatever it says now. |
|
||
| `hale2026harmonic#f929eq9@cd6cdb` | That sentence as it read when its `node_hash` began `cd6cdb`. |
|
||
| `hale2026harmonic#f929eq9@cd6cdb[29:44]` | Code points 29 to 44 of it in that state: "first test case". |
|
||
| `smithlab/hale2026harmonic#f929eq9` | The same sentence in the `smithlab` store. |
|
||
|
||
On the web the ID is the fragment, so a link to `…/smithlab/hale2026harmonic#f929eq9` lands on the sentence.
|
||
See [chapter 6](../spec/06-anchors.md#61-addresses).
|
||
|
||
## Five hashes, five jobs
|
||
|
||
Every node carries five content values.
|
||
|
||
| Value | Covers | Answers |
|
||
| --- | --- | --- |
|
||
| `text_hash` | The text exactly, normalized. | Is this the same quotation, character for character? |
|
||
| `match_hash` | The text with notation folded to one form. | Do two witnesses really agree? |
|
||
| `align_key` | The text folded loosely. | Is this probably the same sentence? |
|
||
| `own_hash` | The node's own content, without its children. | Has this section itself changed, as opposed to something inside it? |
|
||
| `node_hash` | The node and everything below it. | Has anything here changed at all? |
|
||
|
||
The middle two exist because one hash cannot do both of their jobs.
|
||
Chemistry writes the same thing many ways, and every one of these sentences has the same `match_hash`:
|
||
|
||
| As stored | Strict projection |
|
||
| --- | --- |
|
||
| `The solvent is $H_2O$.` | `The solvent is H_{2}O.` |
|
||
| `The solvent is H$_2$O.` | `The solvent is H_{2}O.` |
|
||
| `The solvent is H` and a subscript `2`, then `O.` | `The solvent is H_{2}O.` |
|
||
| `The solvent is H₂O.` | `The solvent is H_{2}O.` |
|
||
| `The solvent is $H_2 O$.` in Typst | `The solvent is H_{2}O.` |
|
||
|
||
But OCR often drops a superscript, and "103" must never be certified as agreeing with 10³.
|
||
So "A rate of 103 per second." and "A rate of 10³ per second." share an `align_key`, which proposes that they are the same sentence, while their `match_hash` values differ, which raises a suspect of kind `number` for a person to resolve.
|
||
The same care keeps "The energy" followed by citation 3 from matching the text "The energy3".
|
||
See [chapter 5](../spec/05-hashes.md) and [vectors/projections.jsonl](../vectors/projections.jsonl).
|
||
|
||
## Anchors
|
||
|
||
An anchor is how anything points into a document: a Lode extract, an annotation, a dataset value, a draft's citation.
|
||
It is a pinned address with enough context to find its text again after the document changes:
|
||
|
||
```json
|
||
{"format":"scienting-anchor/1","store":"smithlab","doc":"hale2026harmonic","node":"f929eq9","kind":"sentence",
|
||
"own_hash":"d5b2eb…","node_hash":"cd6cdb…","span":[29,44],
|
||
"quote":{"exact":"first test case","prefix":"Water, H$_2$O, served as the ","suffix":", with a bending band near 1595 "},
|
||
"rev":12}
|
||
```
|
||
|
||
`rev` is the document's revision when the anchor was made, counted in corrections rather than clock time.
|
||
When the document changes, an anchor resolves one of five ways:
|
||
|
||
| Result | Means | In the example |
|
||
| --- | --- | --- |
|
||
| `current` | Nothing it points at has changed. | A note on the water sentence. |
|
||
| `changed-below` | A container is unchanged in itself, but something inside it changed; a drift figure says how much. | A note on Table 1 after a row was inserted. |
|
||
| `moved` | Its node was merged into another that still holds the quote. | A note on half of a sentence that was wrongly split, then rejoined. |
|
||
| `reattached` | Its node changed, but the quote is still there exactly once, so it is re-pinned. | A note on "treated as zero" in a sentence whose superscript was fixed. |
|
||
| `stale` | It can no longer be trusted; a person must re-anchor it. | A note on the cell that read "3l0.0" before it was corrected to "310.0". |
|
||
|
||
See [chapter 6](../spec/06-anchors.md); the [walkthrough](walkthrough.md#5-what-happened-to-the-notes) shows all five.
|
||
|
||
## Corrections
|
||
|
||
A document is never edited in place.
|
||
Every fix is a correction: an operation with its evidence, its author and a reason, appended to `corrections.jsonl`.
|
||
|
||
```json
|
||
{"format":"scienting-correction/1","index":1,"op":"replace","target":"nkm3xn5",
|
||
"find":"3l0.0","replace":"310.0",
|
||
"evidence":[{"kind":"crop","sha256":"9b6fe6…"}],"author":{"key":"ana"},
|
||
"reason":"OCR read the digit 1 as the letter l.","at":"2026-10-02T10:00:00Z",
|
||
"before":"2fd1e9…","after":"21f13b…","touched":["nkm3xn5"]}
|
||
```
|
||
|
||
The operations are `replace`, `insert`, `delete`, `split`, `merge`, `move`, `reclassify`, `relabel` and `set`.
|
||
A gate refuses any correction that would change a digit without a crop of the page as evidence, or that would let a digit change anywhere it did not declare.
|
||
See [chapter 11](../spec/11-corrections.md).
|
||
|
||
## Fidelity
|
||
|
||
Every leaf has a level that says how far its text was checked.
|
||
|
||
| Level | Means |
|
||
| --- | --- |
|
||
| F0 converted | One reading, not corroborated. |
|
||
| F1 resolved | Every doubt about it was answered against the page image. |
|
||
| F2 corroborated | An independent witness agrees exactly, or the text was born digital. |
|
||
| F3 proofread | A named person compared it with the page and signed for its current text. |
|
||
|
||
A level counts only at the node's current hash, so a correction drops a proofread sentence back until someone signs for it again.
|
||
The example reports 28 leaves at F3, 12 at F2, 4 at F1 and 1 at F0.
|
||
See [chapter 8](../spec/08-fidelity.md).
|
||
|
||
## Receipts and spot-checks
|
||
|
||
Nobody signs three thousand nodes one at a time.
|
||
A reviewer signs a **receipt**: one signature over a Merkle root of everything reviewed in a sitting, with the evidence looked at.
|
||
In the example, the reviewer `ana` signs one receipt for 23 nodes: Table 1's cells, the equation and the sentences around it.
|
||
|
||
Signing once per sitting invites rubber-stamping, so a second reviewer spot-checks a sample.
|
||
The sample is chosen by the checker's Ed25519 signature over the receipt, which neither reviewer can steer, and anyone can recompute it.
|
||
Across samples, the record states a bound on each reviewer's error rate: zero errors in 10 sampled nodes bounds the rate below 26%, and zero errors in 300 bounds it below 1%.
|
||
|
||
A spot-check that finds an error suspends the receipt; a revocation withdraws it.
|
||
Either way the nodes fall back to the level they had without it.
|
||
See [chapter 8](../spec/08-fidelity.md#83-receipts).
|
||
|
||
## Page fingerprints
|
||
|
||
Publishers stamp each download with the institution and time, so two teammates' copies of one paper rarely share a SHA-256.
|
||
Each page therefore also has a fingerprint: a hash of its size, its text with the stamp lines removed, and its images.
|
||
Two downloads of the same page stamped "Downloaded via UNIVERSITY OF EXAMPLE on October 11, 2026" and "Downloaded via EXAMPLE PHARMA INC on March 2, 2027" share the fingerprint `6272e6f3…`; change one digit of the text and it becomes `b545d285…`.
|
||
Geometry and digitizations point at fingerprints, so a re-download keeps every box and every value read off a plot.
|
||
See [chapter 7](../spec/07-geometry.md).
|
||
|
||
## Keeping text private
|
||
|
||
A team may annotate a paper it may not share.
|
||
A **checks-only** package keeps every ID, label, page and box, but withholds the text, and keys every hash, because a plain hash of a short text such as "310.0" is recovered by hashing guesses.
|
||
Inside the team, values are plain SHA-256; exported to another of the team's machines, they are HMACs under the team's key; in a public layer, the key is derived from the paper's page fingerprints, so only someone holding the paper can check them.
|
||
[fixtures/packages/hale2026harmonic.checks-only](../fixtures/packages/hale2026harmonic.checks-only) is the example paper exported this way.
|
||
See [chapter 5.8](../spec/05-hashes.md#58-keyed-hashes).
|
||
|
||
## Rights
|
||
|
||
What a team may do with a paper depends on how it holds the paper.
|
||
Every witness carries a rights record: the license as stated, the basis (an open license, a subscription, a collective license, the team's own work), and which uses that allows, including AI uses by purpose and by where the processing runs.
|
||
Every processing stage declares what it needs and runs only when the rights allow it.
|
||
|
||
The example PDF is held on a subscription that allows text mining and internal AI use on the team's own machines.
|
||
So conversion by a local model is allowed, conversion by an outside service is refused for lacking `ai/internal/outside`, and model review in the team's cloud is refused for lacking `ai/internal/tenancy?` (the question mark means unknown) until a project owner signs a policy that settles it.
|
||
See [chapter 14](../spec/14-rights.md).
|
||
|
||
## Layers and annotations
|
||
|
||
Everything people and models add is an annotation, grouped in layers such as "Novice glosses" or "Journal club, March".
|
||
Annotations are W3C Web Annotations whose selector is an anchor, so any Web Annotation tool can read them.
|
||
This gloss, shortened, explains a phrase for a first-year student:
|
||
|
||
```json
|
||
{"type":"Annotation","motivation":"describing","status":"confirmed",
|
||
"body":{"type":"TextualBody","kind":"gloss","audience":"novice",
|
||
"value":"A *harmonic* term treats a bond like a spring: the energy grows with the square of the stretch."},
|
||
"target":{"type":"SpecificResource","source":"urn:scienting:doc:smithlab/hale2026harmonic",
|
||
"selector":[{"type":"AnchorSelector","anchor":{"node":"sxgcc9h","span":[111,130],"…":"…"}},
|
||
{"type":"TextQuoteSelector","exact":"bond force constant"}]}}
|
||
```
|
||
|
||
One rule decides what is a claim and what is a layer: a statement true independently of the paper belongs in Lode as a checked claim; a statement about reading the paper belongs in a layer.
|
||
Model-written text is always a `generated` annotation, never source.
|
||
See [chapter 18](../spec/18-layers.md).
|
||
|
||
## Exercises
|
||
|
||
Because every sentence and cell has an address, the answer to "find the evidence" is an address, and an address can be graded.
|
||
An exercise asks a student to find the cell giving the Set A bond constant for CT–CT; a student who points at `nkm3xn5` scores 1 of 1 with "That is the cell.", and one who points at the Set B cell scores 0 with "Look in Table 1, row CT–CT."
|
||
Students submit under pseudonyms, and a key whose anchors have gone stale is never used to grade.
|
||
See [chapter 19](../spec/19-exercises.md).
|
||
|
||
## Datasets, digitizations and views
|
||
|
||
Numbers live in prose, in tables and inside figures; all three should end up as data a team can plot and cite.
|
||
A dataset is a Frictionless Data Package with units, uncertainty, and an origin for every value.
|
||
The example's Table 1 becomes a CSV whose every value is anchored to the cell that prints it, digits exactly as printed:
|
||
|
||
```text
|
||
key,bond,set,k_r,r_eq
|
||
ct-ct-a,CT–CT,A,310.0,1.526
|
||
ct-ct-b,CT–CT,B,305.2,1.5260
|
||
```
|
||
|
||
A digitization records how values were read off a plot (axes, calibration, series, uncertainty) and counts only once a person confirms it against an overlay.
|
||
A view is a Vega-Lite specification that names its data by dataset ID and hash.
|
||
See chapters [15](../spec/15-datasets.md), [16](../spec/16-digitization.md) and [17](../spec/17-views.md).
|
||
|
||
## Packages, databases and bundles
|
||
|
||
A document travels as a package: a folder of canonical JSON files.
|
||
|
||
```text
|
||
hale2026harmonic/
|
||
MANIFEST.json every other file, with its SHA-256; its own hash pins the package
|
||
document.json the work, version, metadata, policy, root hash and fidelity
|
||
nodes.jsonl one node per line, in reading order
|
||
witnesses.json each witness and its rights
|
||
pages.jsonl each page and its fingerprint
|
||
locations.jsonl where each node is on each page
|
||
references.jsonl parsed references
|
||
corrections.jsonl every correction, in order
|
||
receipts.jsonl receipts and revocations
|
||
ids.jsonl retired IDs and legacy aliases
|
||
roster.json the keys that signed
|
||
provenance.json tools, versions and where each stage ran
|
||
assets/ images, by SHA-256
|
||
datasets/ datasets reported from the document
|
||
derived/ MathML and other renderings, rebuildable
|
||
```
|
||
|
||
Every line is canonical JSON, so a corrected sentence is a one-line diff, and every file has exactly one valid byte sequence.
|
||
The same data loads into tables that run on SQLite, Dolt and PostgreSQL, and unloads back to the same bytes.
|
||
A bundle is one SQLite file with every table and a full-text index, which a reader or an MCP server can query offline.
|
||
See chapters [12](../spec/12-package.md) and [13](../spec/13-database.md).
|