0 Fork 0

Files

T

feat: zipped packages, target set, checklists, audit fixes and a stress test

Features for the next six months (accessible HTML export, paperpack and
Lode on packages, the personal bundle):

- Zipped packages: the oracle writes and reads `.sct.zip` archives under
  the rules of spec/12 §12.2; validate.py and load.py accept them; a
  byte-identical zip of the tiny package joins the fixtures, and two
  broken archives (traversal, symbolic link) join fixtures/invalid.
- The target set of targeted F3 (spec/08 §8.10) is computed by the
  oracle and pinned by vectors/targets.jsonl under each digit rule; the
  rule now covers cells as well as sentences and the leaves of anchored
  containers, and defines a node's section.
- scienting-checklist/1 (provisional): a conformance check of an export,
  covered by a receipt as ["checklist", id, hash] (spec/08 §8.11), with a
  WCAG 2.1 AA fixture and its receipt.
- QA reports are signed like receipts (spec/08 §8.9).
- A `description` annotation body with role alt or long, so an
  accessible export can find a confirmed figure description; the layer
  fixture carries one, and the store's body-kind constraint allows it.
- scienting-spot-check/1, the seed payload, is registered with a schema.
- exercise, submission and grade are provisional until a course has
  used them.

Audit fixes, each with a vector or a broken package where one was
missing:

- Folder packages: symbolic links are refused on validate and load,
  manifest paths must not collide under case folding, listed files are
  matched by exact name, files are hashed in chunks under a size limit.
- Files are compared byte for byte, so CR LF, a byte order mark, NaN,
  a lone surrogate or invalid UTF-8 is refused instead of translated;
  integers beyond 2^53 are read as doubles, as canonical JSON writes them.
- Receipts: signature.key must equal by and the key must be valid when
  signed; a qa receipt's nodes must be the sample at the original's
  hashes, its scope must name qa.of, its errors must be sampled nodes;
  statuses are computed once, iteratively, so chains of spot-checks
  hundreds deep settle; Base64 and manifest shape errors become refusals
  instead of tracebacks; counts and the fidelity of roster-less packages
  are recomputed.
- The gate enforces rule 11 of spec/11 §11.4: changed nodes are
  validated against the node schema and the references between nodes
  are checked, so a set to a bad value, an insert with unknown
  properties or a delete of a referenced header cell is refused; a
  relabel to null drops the label; a reclassify drops foreign
  attributes; a split strips whitespace across runs; ids.jsonl order
  follows spec/04 §4.4 in every state; a move after itself is refused.
- Anchors: the relabel rule requires that nothing else changed below the
  node; an alias chain ending in a deletion no longer crashes resolve.
- Notation: a script after an empty group applies to the previous atom
  ({}^{14}C); letters are defined by Unicode block in spec/05 §5.4 and
  the oracle; \left) is complex.
- Page fingerprints delete whitespace before folding, so "10 -20" and
  "10-20" fold alike (spec/07 §7.1).
- Grades are summed with exact decimals; the reviewer error bound is
  computed with integer arithmetic.
- receipts and revocations are keyed by (doc_key, id) in SQL, since one
  revocation may be carried by several packages.

Checking:

- tools/stress.py: seeded randomized property tests over canonical JSON,
  normalization, notation, random correction walks, anchor resolution,
  Merkle roots, spot-check samples, error bounds, fingerprints, mutated
  archives and packages, rights and spot-check chains; CI runs it.
- check.py verifies every count and version the prose states against
  the repository, the zipped fixture and crafted archives, the checklist
  coverage, and nine more broken packages.
- CONTRIBUTING.md gains a release checklist.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

2026-10-14 20:07:22 -04:00

3.9 KiB

Raw Permalink Blame History

7. Geometry

7.1 Page fingerprints

Several publishers stamp each download with the institution, date and time, so two copies of one paper rarely share a SHA-256. Every page of a witness therefore also has a fingerprint, scienting-page/1. Geometry and digitizations point at fingerprints, so a re-download keeps every box and every digitization.

The input is the page's size after rotation, its text-layer lines in content-stream order (each a string and, where known, a box), and the SHA-256 of each embedded image's stream data as stored. To compute the fingerprint:

Remove every line that a stamp rule matches (7.2).
Join the remaining lines with U+0020.
Delete every whitespace character and every character of General Category Cc or Cf.
Fold Unicode script characters and dashes before digits as the strict projection does for text (5.4), then apply NFKC.
Round the width and height to whole points, half up.
The fingerprint is SHA-256 of the canonical JSON of {"format": "scienting-page/1", "size": [width, height], "text": the text, "images": the image hashes, sorted}.

Whitespace is deleted rather than collapsed because text extractors disagree about where they insert spaces, and it is deleted before folding so that "10 -20" and "10-20" fold alike. A scanned page with no text layer is fingerprinted by its size and its images alone. Two witnesses whose fingerprints are equal, page for page, are the same edition of the paper.

How lines are extracted from a PDF is outside this text; the vectors give the lines, so implementations are tested on the function. fingerprints.jsonl gives two differently stamped downloads of one ACS-style page and of one Wiley-style page that share their fingerprints, a page with one changed digit that does not, and a scan.

7.2 Stamp rules

Stamp rules are published in scienting-stamps/1 files. Each rule has an id, a pattern, a where (any, or margin with a margin in points, default 36) and a note. A rule matches a line when its pattern is found in the line's normalized text, with ASCII letters compared without case; a margin rule matches only lines whose box lies wholly within the margin of one edge. Patterns use only syntax that ECMAScript and Python read alike: literals, ., *, +, ?, ^, $, classes and escaped punctuation.

registry/stamps/scienting-stamps-1.json holds the published rules. A team adds rules it learns, when two witnesses of one version differ only in such lines, in files of its own with learned: true. A missed stamp splits one edition into two; an over-broad rule hides a real difference.

7.3 `pages.jsonl`

Property	Holds
`witness`	The witness ID.
`page`	The page number in the witness, from 1.
`page_fp`	The fingerprint.
`size`	`[width, height]` in points, after rotation.
`page_label`	The printed page label, when there is one.
`text_layer`	Whether the page has a text layer.
`stamps_removed`	The IDs of the rules that removed lines, in line order, when any did.

Pages sort by witness ID, then page.

7.4 `locations.jsonl`

Property	Holds
`node`	The node ID.
`page_fp`	The page's fingerprint, which some page of some witness has.
`page`	The page number.
`page_label`	The printed label.
`boxes`	One or more `[x0, y0, x1, y1]` in PDF points, origin at the top left after rotation.
`approx`	`true` when the boxes come from OCR line boxes rather than glyphs.

A node has one record per page it appears on; a sentence that wraps has several boxes. Locations sort by the node's position in document order, then page, then fingerprint. Because locations point at fingerprinted pages, a reader can rebuild a document from its own download: it fingerprints the pages, finds the matching ones and reads the text at the stored boxes.

3.9 KiB Raw Permalink Blame History