feat: alignment pairs a misread sentence with its other reading; a suspect names its witness

spec/05 §5.6 gains a substitution step: an unpaired sentence beside one at
least 0.8 alike (2L/(|a|+|b|) over the lossy projections' code points) is
paired with it, so a misread is one number, equation or word suspect rather
than a dropped and an added one. §5.7 and suspect.schema.json gain an
optional `witness`, checked by doc.suspect.witness. alignment.jsonl gains a
`substitutions` case; the first case is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
alexmandClaude committed 2026-10-16 22:41:15 -04:00
1 parent 9e24ca2306
commit 3836a6c885
9 files changed
+104 -7

No files matched your search

+12 -5
View File
@@ -147,20 +147,27 @@ Each reading is a list of sentences in document order, each with its strict proj
Take the candidates in the order of the first reading, and choose the longest subsequence whose positions in the second reading increase; among equally long ones, the one whose sequence of positions in the first reading is lexicographically smallest.
2. **Between anchors**, and before the first and after the last, align by the longest common subsequence of lossy projections.
Ties are broken deterministically: walking both segments from their start, pair two sentences whose lossy projections are equal; otherwise skip the first reading's sentence when the remainder without it has a common subsequence at least as long as without the second's, and skip the second's otherwise.
3. **Compare** each pair.
3. **Substitutions.** A sentence one reading misread has a different lossy projection, so steps 1 and 2 leave it unpaired on both sides; this step pairs it with its other reading.
Steps 1 and 2 leave stretches between consecutive pairs, and before the first and after the last, with unpaired sentences on either side.
In each stretch, take the first reading's unpaired sentences in order, and pair each whose prose length is at least 20 with the first of the second reading's unpaired sentences in the same stretch that comes after the last one paired in this step, has a prose length of at least 20, and has a similarity to it of at least 0.8.
The similarity of two lossy projections *a* and *b* is 2*L* / (|*a*| + |*b*|), where *L* is the length of their longest common subsequence of code points and |*a*| and |*b*| are their lengths in code points.
A pair made here never agrees, since its lossy projections differ, so step 4 makes it a `number`, `equation` or `word` suspect rather than a `dropped` and an `added` one.
4. **Compare** each pair.
They agree when their strict projections and complex keys are equal.
Otherwise the pair is a suspect: of kind `number` when their numeric skeletons differ, else `equation` when their complex keys differ, else `word`.
The numeric skeleton keeps only digits, U+2212, `.`, `,`, `+`, `_{`, `^{` and `}`.
4. **Structure.** A pair whose contexts differ is also a `structure` suspect; two consecutive pairs that share a container in one reading but not in the other are a `join` suspect.
5. **Unplaced** sentences are `dropped` (in the first reading only) or `added` (in the second only).
5. **Structure.** A pair whose contexts differ is also a `structure` suspect; two consecutive pairs that share a container in one reading but not in the other are a `join` suspect.
6. **Unplaced** sentences are `dropped` (in the first reading only) or `added` (in the second only).
Pairs are listed in the first reading's order, whichever step made them.
Paired sentences keep the first reading's ID even when they disagree, so a reading that lost a superscript is caught rather than accepted.
`alignment.jsonl` aligns two readings that exercise every rule, including two prose-thin sentences aligned by their order.
`alignment.jsonl` aligns two readings that exercise every rule, including two prose-thin sentences aligned by their order, and a second pair of readings with substitutions: a misread number, a misread word, a sentence too short to be paired this way, and two sentences too unlike to be one.
## 5.7 Suspects
A suspect, `scienting-suspect/1`, records a doubtful reading for review: its `id`, its `kind` (`word`, `number`, `equation`, `structure`, `join`, `added`, `dropped`, `boundary` for an uncertain sentence boundary, or `reading` for anything else), the node, the readings compared, an optional crop by SHA-256 and a note.
Suspects may travel in a package's `suspects.jsonl`, sorted by ID, until they are resolved.
When the second reading, `b`, is another witness's reading of the node (a PDF's text layer, a JATS file, a LaTeX source), `witness` names that witness by its ID in `witnesses.json`, so a reviewer knows what the node was compared with.
Suspects may travel in a package's `suspects.jsonl`, sorted by ID, until they are resolved; in a package, a suspect's `witness` is one of the package's witnesses.
## 5.8 Keyed hashes