alexmandClaude
ea0a7efa78
Features for the next six months (accessible HTML export, paperpack and
Lode on packages, the personal bundle):
- Zipped packages: the oracle writes and reads `.sct.zip` archives under
the rules of spec/12 §12.2; validate.py and load.py accept them; a
byte-identical zip of the tiny package joins the fixtures, and two
broken archives (traversal, symbolic link) join fixtures/invalid.
- The target set of targeted F3 (spec/08 §8.10) is computed by the
oracle and pinned by vectors/targets.jsonl under each digit rule; the
rule now covers cells as well as sentences and the leaves of anchored
containers, and defines a node's section.
- scienting-checklist/1 (provisional): a conformance check of an export,
covered by a receipt as ["checklist", id, hash] (spec/08 §8.11), with a
WCAG 2.1 AA fixture and its receipt.
- QA reports are signed like receipts (spec/08 §8.9).
- A `description` annotation body with role alt or long, so an
accessible export can find a confirmed figure description; the layer
fixture carries one, and the store's body-kind constraint allows it.
- scienting-spot-check/1, the seed payload, is registered with a schema.
- exercise, submission and grade are provisional until a course has
used them.
Audit fixes, each with a vector or a broken package where one was
missing:
- Folder packages: symbolic links are refused on validate and load,
manifest paths must not collide under case folding, listed files are
matched by exact name, files are hashed in chunks under a size limit.
- Files are compared byte for byte, so CR LF, a byte order mark, NaN,
a lone surrogate or invalid UTF-8 is refused instead of translated;
integers beyond 2^53 are read as doubles, as canonical JSON writes them.
- Receipts: signature.key must equal by and the key must be valid when
signed; a qa receipt's nodes must be the sample at the original's
hashes, its scope must name qa.of, its errors must be sampled nodes;
statuses are computed once, iteratively, so chains of spot-checks
hundreds deep settle; Base64 and manifest shape errors become refusals
instead of tracebacks; counts and the fidelity of roster-less packages
are recomputed.
- The gate enforces rule 11 of spec/11 §11.4: changed nodes are
validated against the node schema and the references between nodes
are checked, so a set to a bad value, an insert with unknown
properties or a delete of a referenced header cell is refused; a
relabel to null drops the label; a reclassify drops foreign
attributes; a split strips whitespace across runs; ids.jsonl order
follows spec/04 §4.4 in every state; a move after itself is refused.
- Anchors: the relabel rule requires that nothing else changed below the
node; an alias chain ending in a deletion no longer crashes resolve.
- Notation: a script after an empty group applies to the previous atom
({}^{14}C); letters are defined by Unicode block in spec/05 §5.4 and
the oracle; \left) is complex.
- Page fingerprints delete whitespace before folding, so "10 -20" and
"10-20" fold alike (spec/07 §7.1).
- Grades are summed with exact decimals; the reviewer error bound is
computed with integer arithmetic.
- receipts and revocations are keyed by (doc_key, id) in SQL, since one
revocation may be carried by several packages.
Checking:
- tools/stress.py: seeded randomized property tests over canonical JSON,
normalization, notation, random correction walks, anchor resolution,
Merkle roots, spot-check samples, error bounds, fingerprints, mutated
archives and packages, rights and spot-check chains; CI runs it.
- check.py verifies every count and version the prose states against
the repository, the zipped fixture and crafted archives, the checklist
coverage, and nine more broken packages.
- CONTRIBUTING.md gains a release checklist.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
65 lines
3.9 KiB
Markdown
65 lines
3.9 KiB
Markdown
# 7. Geometry
|
|
|
|
## 7.1 Page fingerprints
|
|
|
|
Several publishers stamp each download with the institution, date and time, so two copies of one paper rarely share a SHA-256.
|
|
Every page of a witness therefore also has a fingerprint, `scienting-page/1`.
|
|
Geometry and digitizations point at fingerprints, so a re-download keeps every box and every digitization.
|
|
|
|
The input is the page's size after rotation, its text-layer lines in content-stream order (each a string and, where known, a box), and the SHA-256 of each embedded image's stream data as stored.
|
|
To compute the fingerprint:
|
|
|
|
1. Remove every line that a stamp rule matches ([7.2](#72-stamp-rules)).
|
|
2. Join the remaining lines with U+0020.
|
|
3. Delete every whitespace character and every character of General Category Cc or Cf.
|
|
4. Fold Unicode script characters and dashes before digits as the strict projection does for text ([5.4](05-hashes.md#54-simple-notation)), then apply NFKC.
|
|
5. Round the width and height to whole points, half up.
|
|
6. The fingerprint is SHA-256 of the canonical JSON of `{"format": "scienting-page/1", "size": [width, height], "text": the text, "images": the image hashes, sorted}`.
|
|
|
|
Whitespace is deleted rather than collapsed because text extractors disagree about where they insert spaces, and it is deleted before folding so that "10 -20" and "10-20" fold alike.
|
|
A scanned page with no text layer is fingerprinted by its size and its images alone.
|
|
Two witnesses whose fingerprints are equal, page for page, are the same edition of the paper.
|
|
|
|
How lines are extracted from a PDF is outside this text; the vectors give the lines, so implementations are tested on the function.
|
|
`fingerprints.jsonl` gives two differently stamped downloads of one ACS-style page and of one Wiley-style page that share their fingerprints, a page with one changed digit that does not, and a scan.
|
|
|
|
## 7.2 Stamp rules
|
|
|
|
Stamp rules are published in `scienting-stamps/1` files.
|
|
Each rule has an `id`, a `pattern`, a `where` (`any`, or `margin` with a `margin` in points, default 36) and a note.
|
|
A rule matches a line when its pattern is found in the line's normalized text, with ASCII letters compared without case; a `margin` rule matches only lines whose box lies wholly within the margin of one edge.
|
|
Patterns use only syntax that ECMAScript and Python read alike: literals, `.`, `*`, `+`, `?`, `^`, `$`, classes and escaped punctuation.
|
|
|
|
[registry/stamps/scienting-stamps-1.json](../registry/stamps/scienting-stamps-1.json) holds the published rules.
|
|
A team adds rules it learns, when two witnesses of one version differ only in such lines, in files of its own with `learned: true`.
|
|
A missed stamp splits one edition into two; an over-broad rule hides a real difference.
|
|
|
|
## 7.3 `pages.jsonl`
|
|
|
|
| Property | Holds |
|
|
| --- | --- |
|
|
| `witness` | The witness ID. |
|
|
| `page` | The page number in the witness, from 1. |
|
|
| `page_fp` | The fingerprint. |
|
|
| `size` | `[width, height]` in points, after rotation. |
|
|
| `page_label` | The printed page label, when there is one. |
|
|
| `text_layer` | Whether the page has a text layer. |
|
|
| `stamps_removed` | The IDs of the rules that removed lines, in line order, when any did. |
|
|
|
|
Pages sort by witness ID, then page.
|
|
|
|
## 7.4 `locations.jsonl`
|
|
|
|
| Property | Holds |
|
|
| --- | --- |
|
|
| `node` | The node ID. |
|
|
| `page_fp` | The page's fingerprint, which some page of some witness has. |
|
|
| `page` | The page number. |
|
|
| `page_label` | The printed label. |
|
|
| `boxes` | One or more `[x0, y0, x1, y1]` in PDF points, origin at the top left after rotation. |
|
|
| `approx` | `true` when the boxes come from OCR line boxes rather than glyphs. |
|
|
|
|
A node has one record per page it appears on; a sentence that wraps has several boxes.
|
|
Locations sort by the node's position in document order, then page, then fingerprint.
|
|
Because locations point at fingerprinted pages, a reader can rebuild a document from its own download: it fingerprints the pages, finds the matching ones and reads the text at the stored boxes.
|