feat: alignment pairs a misread sentence with its other reading; a suspect names its witness

spec/05 §5.6 gains a substitution step: an unpaired sentence beside one at
least 0.8 alike (2L/(|a|+|b|) over the lossy projections' code points) is
paired with it, so a misread is one number, equation or word suspect rather
than a dropped and an added one. §5.7 and suspect.schema.json gain an
optional `witness`, checked by doc.suspect.witness. alignment.jsonl gains a
`substitutions` case; the first case is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

CHANGELOG.md

@@ -35,3 +35,14 @@ Packaging, commands and lazy imports only; the formats stay at 0.1.0, and no com
 - `src/` is Apache-2.0, as `tools/` was, and the package metadata states `Apache-2.0 AND CC0-1.0` for the code and the data it carries.
 - The pixi workspace installs the package editable, and a `dev` environment builds the sdist and the wheel with `pixi run -e dev dist`.
 - Pushing a version tag publishes the package to the Gitea package registry through `.gitea/workflows/publish.yml` ([deploy/](deploy/README.md#publish-the-python-package)); a release tag versions the package, and [CONTRIBUTING.md](CONTRIBUTING.md#releasing) says how that relates to the format version.
+
+### Unreleased
+
+Two changes asked for by the first producer that compares a converted reading with a second witness (paperstract with a JATS file).
+Both are minor changes in the sense of [spec/00](spec/00-introduction.md): no valid file becomes invalid, and no hash, ID or key changes; the next release numbers them.
+
+- Alignment has a substitution step ([spec/05 §5.6](spec/05-hashes.md#56-alignment), step 3): in each stretch between pairs, an unpaired sentence of the first reading is paired with the first unpaired sentence of the second whose similarity (2*L* / (|*a*| + |*b*|) over code points of the lossy projections, *L* their longest common subsequence) is at least 0.8, both of prose length 20 or more.
+  A sentence one reading misread is now one `number`, `equation` or `word` suspect rather than a `dropped` and an `added` one, so a producer re-aligning the same two readings can get different suspects than before; the pairs and suspects of readings with no such sentence are unchanged, as the first alignment vector is.
+  `paperspec.align` gains `lcs_length` (bit-parallel, Hyyrö 2004) and `similarity`, and `alignment.jsonl` a second case, `substitutions`: a misread number, a misread word, a sentence too short to be paired this way and two sentences too unlike to be one.
+- A suspect may name the witness its second reading came from (`witness`, a witness ID; [spec/05 §5.7](spec/05-hashes.md#57-suspects), `suspect.schema.json`), and a package whose suspect names a witness it does not have breaks `doc.suspect.witness`.
+  No broken package exercises the rule yet: adding one changes the count of broken packages that the 0.1.0 entry above states, so it waits for the release.

registry/rules.json

@@ -363,6 +363,11 @@
       "description": "A same-document xref names no node.",
       "chapter": "spec/22-conformance.md"
     },
+    {
+      "id": "doc.suspect.witness",
+      "description": "A suspect names a witness the package does not have.",
+      "chapter": "spec/22-conformance.md"
+    },
     {
       "id": "doc.table.header",
       "description": "A column or row header path names a node that is not a cell.",

schemas/suspect.schema.json

@@ -60,6 +60,9 @@
         }
       ]
     },
+    "witness": {
+      "$ref": "common.schema.json#/$defs/nodeId"
+    },
     "crop": {
       "$ref": "common.schema.json#/$defs/hash"
     },

spec/05-hashes.md

@@ -147,20 +147,27 @@ Each reading is a list of sentences in document order, each with its strict proj
    Take the candidates in the order of the first reading, and choose the longest subsequence whose positions in the second reading increase; among equally long ones, the one whose sequence of positions in the first reading is lexicographically smallest.
 2. **Between anchors**, and before the first and after the last, align by the longest common subsequence of lossy projections.
    Ties are broken deterministically: walking both segments from their start, pair two sentences whose lossy projections are equal; otherwise skip the first reading's sentence when the remainder without it has a common subsequence at least as long as without the second's, and skip the second's otherwise.
-3. **Compare** each pair.
+3. **Substitutions.** A sentence one reading misread has a different lossy projection, so steps 1 and 2 leave it unpaired on both sides; this step pairs it with its other reading.
+   Steps 1 and 2 leave stretches between consecutive pairs, and before the first and after the last, with unpaired sentences on either side.
+   In each stretch, take the first reading's unpaired sentences in order, and pair each whose prose length is at least 20 with the first of the second reading's unpaired sentences in the same stretch that comes after the last one paired in this step, has a prose length of at least 20, and has a similarity to it of at least 0.8.
+   The similarity of two lossy projections *a* and *b* is 2*L* / (|*a*| + |*b*|), where *L* is the length of their longest common subsequence of code points and |*a*| and |*b*| are their lengths in code points.
+   A pair made here never agrees, since its lossy projections differ, so step 4 makes it a `number`, `equation` or `word` suspect rather than a `dropped` and an `added` one.
+4. **Compare** each pair.
    They agree when their strict projections and complex keys are equal.
    Otherwise the pair is a suspect: of kind `number` when their numeric skeletons differ, else `equation` when their complex keys differ, else `word`.
    The numeric skeleton keeps only digits, U+2212, `.`, `,`, `+`, `_{`, `^{` and `}`.
-4. **Structure.** A pair whose contexts differ is also a `structure` suspect; two consecutive pairs that share a container in one reading but not in the other are a `join` suspect.
+5. **Structure.** A pair whose contexts differ is also a `structure` suspect; two consecutive pairs that share a container in one reading but not in the other are a `join` suspect.
-5. **Unplaced** sentences are `dropped` (in the first reading only) or `added` (in the second only).
+6. **Unplaced** sentences are `dropped` (in the first reading only) or `added` (in the second only).
 
+Pairs are listed in the first reading's order, whichever step made them.
 Paired sentences keep the first reading's ID even when they disagree, so a reading that lost a superscript is caught rather than accepted.
-`alignment.jsonl` aligns two readings that exercise every rule, including two prose-thin sentences aligned by their order.
+`alignment.jsonl` aligns two readings that exercise every rule, including two prose-thin sentences aligned by their order, and a second pair of readings with substitutions: a misread number, a misread word, a sentence too short to be paired this way, and two sentences too unlike to be one.
 
 ## 5.7 Suspects
 
 A suspect, `scienting-suspect/1`, records a doubtful reading for review: its `id`, its `kind` (`word`, `number`, `equation`, `structure`, `join`, `added`, `dropped`, `boundary` for an uncertain sentence boundary, or `reading` for anything else), the node, the readings compared, an optional crop by SHA-256 and a note.
-Suspects may travel in a package's `suspects.jsonl`, sorted by ID, until they are resolved.
+When the second reading, `b`, is another witness's reading of the node (a PDF's text layer, a JATS file, a LaTeX source), `witness` names that witness by its ID in `witnesses.json`, so a reviewer knows what the node was compared with.
+Suspects may travel in a package's `suspects.jsonl`, sorted by ID, until they are resolved; in a package, a suspect's `witness` is one of the package's witnesses.
 
 ## 5.8 Keyed hashes
 

src/paperspec/align.py

@@ -9,6 +9,7 @@ import unicodedata
 from .hashing import prose_length
 
 MIN_PROSE = 20
+MIN_SIMILAR = 0.8
 
 
 def _numeric_skeleton(strict):
@@ -82,6 +83,56 @@ def _lcs(A, B, a0, a1, b0, b1):
     return out
 
 
+def lcs_length(a, b):
+    """Length of the longest common subsequence of two strings' code points (bit-parallel, Hyyrö 2004)."""
+    if len(a) < len(b):
+        a, b = b, a
+    masks = {}
+    for i, ch in enumerate(a):
+        masks[ch] = masks.get(ch, 0) | (1 << i)
+    full = (1 << len(a)) - 1
+    v = full
+    for ch in b:
+        u = v & masks.get(ch, 0)
+        v = ((v + u) | (v - u)) & full
+    return len(a) - v.bit_count()
+
+
+def similarity(a, b):
+    """2L / (|a| + |b|) over code points, L the longest common subsequence; 1 for two empty strings."""
+    if not a and not b:
+        return 1.0
+    return 2 * lcs_length(a, b) / (len(a) + len(b))
+
+
+def _similar(a, b):
+    """Whether two lossy projections are at least MIN_SIMILAR alike; exact, with two cheap upper bounds first."""
+    total = len(a) + len(b)
+    if 2 * min(len(a), len(b)) < MIN_SIMILAR * total:
+        return False
+    common = sum(min(n, b.count(ch)) for ch, n in {ch: a.count(ch) for ch in set(a)}.items())
+    if 2 * common < MIN_SIMILAR * total:
+        return False
+    return 2 * lcs_length(a, b) >= MIN_SIMILAR * total
+
+
+def _substitutions(A, B, pairs):
+    """Step 3: in each stretch between pairs, an unpaired sentence beside a nearly equal one is paired with it."""
+    out = []
+    bounds = [(-1, -1)] + pairs + [(len(A), len(B))]
+    for (a0, b0), (a1, b1) in zip(bounds, bounds[1:]):
+        nxt = b0 + 1
+        for i in range(a0 + 1, a1):
+            if prose_length(A[i]["lossy"]) < MIN_PROSE:
+                continue
+            for j in range(nxt, b1):
+                if prose_length(B[j]["lossy"]) >= MIN_PROSE and _similar(A[i]["lossy"], B[j]["lossy"]):
+                    out.append((i, j))
+                    nxt = j + 1
+                    break
+    return out
+
+
 def align(A, B):
     """Align reading B to reading A. Returns {"pairs": [...], "suspects": [...]}."""
     count_a, count_b = {}, {}
@@ -102,6 +153,7 @@ def align(A, B):
         if ai < len(A):
             pairs.append((ai, bi))
         prev_a, prev_b = ai + 1, bi + 1
+    pairs = sorted(pairs + _substitutions(A, B, pairs))
     out_pairs, suspects = [], []
     for ai, bi in pairs:
         kind = classify(A[ai], B[bi])

src/paperspec/cli/build.py

@@ -1126,7 +1126,22 @@ def vectors(out, hale, tiny):
          item("b5", [T("where "), M("\\frac{c}{e}"), T(" is the bond length.")], "q2"),
          item("b6", [T("A sentence only the second reading has, also long.")], "q2"),
          item("b7", [T("The last sentence of the section is unique and long enough.")], "q2")]
-    wv(out, "alignment.jsonl", [{"id": "two-readings", "a": A, "b": B, "expect": AL.align(A, B)}])
+    # Substitutions (step 3): a misread number and a misread word pair with their other readings; a short
+    # sentence and two sentences too unlike to be one stay dropped and added.
+    SA = [item("c1", [T("The force constants were fitted to ab initio scans of every bond.")], "p1"),
+          item("c2", [T("The bond constant was 310.0 kcal/mol for every aliphatic carbon.")], "p1"),
+          item("c3", [T("This prototypal set was later extended to nucleic acids.")], "p1"),
+          item("c4", [T("See below.")], "p1"),
+          item("c5", [T("Torsions were refit against a second, larger set of conformers.")], "p1"),
+          item("c6", [T("The charges came from a two-stage restrained fit to the potential.")], "p2")]
+    SB = [item("d1", [T("The force constants were fitted to ab initio scans of every bond.")], "q1"),
+          item("d2", [T("The bond constant was 3l0.0 kcal/mol for every aliphatic carbon.")], "q1"),
+          item("d3", [T("This prototypical set was later extended to nucleic acids.")], "q1"),
+          item("d4", [T("See bclow.")], "q1"),
+          item("d5", [T("Only the second reading prints this line about water models.")], "q1"),
+          item("d6", [T("The charges came from a two-stage restrained fit to the potential.")], "q2")]
+    wv(out, "alignment.jsonl", [{"id": "two-readings", "a": A, "b": B, "expect": AL.align(A, B)},
+                                {"id": "substitutions", "a": SA, "b": SB, "expect": AL.align(SA, SB)}])
     # Corrections, op by op, on the tiny document.
     wv(out, "corrections.jsonl", corrections_vectors())
     # Anchor resolution against the hale package.

src/paperspec/package.py

@@ -401,6 +401,9 @@ def _validate(root, store, key, rep, limit=SIZE_LIMIT):
     sorder = sorted(data["suspects.jsonl"], key=lambda x: x["id"])
     if sorder != data["suspects.jsonl"]:
         rep.add("doc.order.suspects", "suspects.jsonl", "suspects sort by ID")
+    for s in data["suspects.jsonl"]:
+        if "witness" in s and s["witness"] not in wits:
+            rep.add("doc.suspect.witness", s["id"], "a suspect's witness is one of the package's witnesses")
 
     # Receipts and fidelity.
     roster = {k["id"]: k for k in data.get("roster.json", {}).get("keys", [])}

vectors/README.md

@@ -138,7 +138,7 @@ A candidate format changes only with a recorded reason and a new or changed vect
 
 A passing run shows one thing: for these inputs, your implementation computes the same values as the reference implementation in `src/paperspec/`.
 That code computed the expected outputs, and `pixi run build` regenerates them ([tools/](../tools/README.md)).
-The vectors hold 311 cases in all, some of them small: `hashes.jsonl` has one document, `alignment.jsonl` one pair of readings, `offsets.jsonl` two cases.
+The vectors hold 312 cases in all, some of them small: `hashes.jsonl` has one document, `alignment.jsonl` two pairs of readings, `offsets.jsonl` two cases.
 
 The vectors do not show the following.
 

vectors/alignment.jsonl

@@ -1 +1,2 @@
 {"a":[{"complex":[],"container":"h","context":"heading","id":"a1","lossy":"methods","runs":[{"kind":"text","text":"Methods"}],"strict":"Methods"},{"complex":[],"container":"p1","context":"paragraph","id":"a2","lossy":"everybondwasscannedinstepsof0.01åarounditsminimum.","runs":[{"kind":"text","text":"Every bond was scanned in steps of 0.01 Å around its minimum."}],"strict":"Every bond was scanned in steps of 0.01 Å around its minimum."},{"complex":["tex:\\frac{a}{b}"],"container":"p1","context":"paragraph","id":"a3","lossy":"whereisthebondlength.","runs":[{"kind":"text","text":"where "},{"kind":"math","notation":"tex","source":"\\frac{a}{b}"},{"kind":"text","text":" is the bond length."}],"strict":"where  is the bond length."},{"complex":[],"container":"p1","context":"paragraph","id":"a4","lossy":"thefittedrateis103persecondatroomtemperature.","runs":[{"kind":"text","text":"The fitted rate is 10³ per second at room temperature."}],"strict":"The fitted rate is 10^{3} per second at room temperature."},{"complex":["tex:\\frac{c}{d}"],"container":"p2","context":"paragraph","id":"a5","lossy":"whereisthebondlength.","runs":[{"kind":"text","text":"where "},{"kind":"math","notation":"tex","source":"\\frac{c}{d}"},{"kind":"text","text":" is the bond length."}],"strict":"where  is the bond length."},{"complex":[],"container":"p2","context":"paragraph","id":"a6","lossy":"thissentencewasdroppedbythesecondreadingentirely.","runs":[{"kind":"text","text":"This sentence was dropped by the second reading entirely."}],"strict":"This sentence was dropped by the second reading entirely."},{"complex":[],"container":"p2","context":"paragraph","id":"a7","lossy":"thelastsentenceofthesectionisuniqueandlongenough.","runs":[{"kind":"text","text":"The last sentence of the section is unique and long enough."}],"strict":"The last sentence of the section is unique and long enough."}],"b":[{"complex":[],"container":"x","context":"paragraph","id":"b1","lossy":"methods","runs":[{"kind":"text","text":"Methods"}],"strict":"Methods"},{"complex":[],"container":"q1","context":"paragraph","id":"b2","lossy":"everybondwasscannedinstepsof0.01åarounditsminimum.","runs":[{"kind":"text","text":"Every bond was scanned in steps of 0.01 Å around its minimum."}],"strict":"Every bond was scanned in steps of 0.01 Å around its minimum."},{"complex":["tex:\\frac{a}{b}"],"container":"q1","context":"paragraph","id":"b3","lossy":"whereisthebondlength.","runs":[{"kind":"text","text":"where "},{"kind":"math","notation":"tex","source":"\\frac{a}{b}"},{"kind":"text","text":" is the bond length."}],"strict":"where  is the bond length."},{"complex":[],"container":"q2","context":"paragraph","id":"b4","lossy":"thefittedrateis103persecondatroomtemperature.","runs":[{"kind":"text","text":"The fitted rate is 103 per second at room temperature."}],"strict":"The fitted rate is 103 per second at room temperature."},{"complex":["tex:\\frac{c}{e}"],"container":"q2","context":"paragraph","id":"b5","lossy":"whereisthebondlength.","runs":[{"kind":"text","text":"where "},{"kind":"math","notation":"tex","source":"\\frac{c}{e}"},{"kind":"text","text":" is the bond length."}],"strict":"where  is the bond length."},{"complex":[],"container":"q2","context":"paragraph","id":"b6","lossy":"asentenceonlythesecondreadinghas,alsolong.","runs":[{"kind":"text","text":"A sentence only the second reading has, also long."}],"strict":"A sentence only the second reading has, also long."},{"complex":[],"container":"q2","context":"paragraph","id":"b7","lossy":"thelastsentenceofthesectionisuniqueandlongenough.","runs":[{"kind":"text","text":"The last sentence of the section is unique and long enough."}],"strict":"The last sentence of the section is unique and long enough."}],"expect":{"pairs":[{"a":"a1","agree":true,"b":"b1"},{"a":"a2","agree":true,"b":"b2"},{"a":"a3","agree":true,"b":"b3"},{"a":"a4","agree":false,"b":"b4"},{"a":"a5","agree":false,"b":"b5"},{"a":"a7","agree":true,"b":"b7"}],"suspects":[{"a":"a1","b":"b1","kind":"structure"},{"a":"a4","b":"b4","kind":"number"},{"a":"a5","b":"b5","kind":"equation"},{"a":"a4","b":"b4","kind":"join"},{"a":"a5","b":"b5","kind":"join"},{"a":"a6","b":null,"kind":"dropped"},{"a":null,"b":"b6","kind":"added"}]},"id":"two-readings"}
+{"a":[{"complex":[],"container":"p1","context":"paragraph","id":"c1","lossy":"theforceconstantswerefittedtoabinitioscansofeverybond.","runs":[{"kind":"text","text":"The force constants were fitted to ab initio scans of every bond."}],"strict":"The force constants were fitted to ab initio scans of every bond."},{"complex":[],"container":"p1","context":"paragraph","id":"c2","lossy":"thebondconstantwas310.0kcal/molforeveryaliphaticcarbon.","runs":[{"kind":"text","text":"The bond constant was 310.0 kcal/mol for every aliphatic carbon."}],"strict":"The bond constant was 310.0 kcal/mol for every aliphatic carbon."},{"complex":[],"container":"p1","context":"paragraph","id":"c3","lossy":"thisprototypalsetwaslaterextendedtonucleicacids.","runs":[{"kind":"text","text":"This prototypal set was later extended to nucleic acids."}],"strict":"This prototypal set was later extended to nucleic acids."},{"complex":[],"container":"p1","context":"paragraph","id":"c4","lossy":"seebelow.","runs":[{"kind":"text","text":"See below."}],"strict":"See below."},{"complex":[],"container":"p1","context":"paragraph","id":"c5","lossy":"torsionswererefitagainstasecond,largersetofconformers.","runs":[{"kind":"text","text":"Torsions were refit against a second, larger set of conformers."}],"strict":"Torsions were refit against a second, larger set of conformers."},{"complex":[],"container":"p2","context":"paragraph","id":"c6","lossy":"thechargescamefromatwostagerestrainedfittothepotential.","runs":[{"kind":"text","text":"The charges came from a two-stage restrained fit to the potential."}],"strict":"The charges came from a two-stage restrained fit to the potential."}],"b":[{"complex":[],"container":"q1","context":"paragraph","id":"d1","lossy":"theforceconstantswerefittedtoabinitioscansofeverybond.","runs":[{"kind":"text","text":"The force constants were fitted to ab initio scans of every bond."}],"strict":"The force constants were fitted to ab initio scans of every bond."},{"complex":[],"container":"q1","context":"paragraph","id":"d2","lossy":"thebondconstantwas3l0.0kcal/molforeveryaliphaticcarbon.","runs":[{"kind":"text","text":"The bond constant was 3l0.0 kcal/mol for every aliphatic carbon."}],"strict":"The bond constant was 3l0.0 kcal/mol for every aliphatic carbon."},{"complex":[],"container":"q1","context":"paragraph","id":"d3","lossy":"thisprototypicalsetwaslaterextendedtonucleicacids.","runs":[{"kind":"text","text":"This prototypical set was later extended to nucleic acids."}],"strict":"This prototypical set was later extended to nucleic acids."},{"complex":[],"container":"q1","context":"paragraph","id":"d4","lossy":"seebclow.","runs":[{"kind":"text","text":"See bclow."}],"strict":"See bclow."},{"complex":[],"container":"q1","context":"paragraph","id":"d5","lossy":"onlythesecondreadingprintsthislineaboutwatermodels.","runs":[{"kind":"text","text":"Only the second reading prints this line about water models."}],"strict":"Only the second reading prints this line about water models."},{"complex":[],"container":"q2","context":"paragraph","id":"d6","lossy":"thechargescamefromatwostagerestrainedfittothepotential.","runs":[{"kind":"text","text":"The charges came from a two-stage restrained fit to the potential."}],"strict":"The charges came from a two-stage restrained fit to the potential."}],"expect":{"pairs":[{"a":"c1","agree":true,"b":"d1"},{"a":"c2","agree":false,"b":"d2"},{"a":"c3","agree":false,"b":"d3"},{"a":"c6","agree":true,"b":"d6"}],"suspects":[{"a":"c2","b":"d2","kind":"number"},{"a":"c3","b":"d3","kind":"word"},{"a":"c4","b":null,"kind":"dropped"},{"a":"c5","b":null,"kind":"dropped"},{"a":null,"b":"d4","kind":"added"},{"a":null,"b":"d5","kind":"added"}]},"id":"substitutions"}