Initial commit

spec/07-geometry.md

@@ -0,0 +1,64 @@
+# 7. Geometry
+
+## 7.1 Page fingerprints
+
+Several publishers stamp each download with the institution, date and time, so two copies of one paper rarely share a SHA-256.
+Every page of a witness therefore also has a fingerprint, `scienting-page/1`.
+Geometry and digitizations point at fingerprints, so a re-download keeps every box and every digitization.
+
+The input is the page's size after rotation, its text-layer lines in content-stream order (each a string and, where known, a box), and the SHA-256 of each embedded image's stream data as stored.
+To compute the fingerprint:
+
+1. Remove every line that a stamp rule matches ([7.2](#72-stamp-rules)).
+2. Join the remaining lines with U+0020.
+3. Fold Unicode script characters and dashes before digits as the strict projection does for text ([5.4](05-hashes.md#54-simple-notation)), then apply NFKC.
+4. Delete every whitespace character and every character of General Category Cc or Cf.
+5. Round the width and height to whole points, half up.
+6. The fingerprint is SHA-256 of the canonical JSON of `{"format": "scienting-page/1", "size": [width, height], "text": the text, "images": the image hashes, sorted}`.
+
+Whitespace is deleted rather than collapsed because text extractors disagree about where they insert spaces.
+A scanned page with no text layer is fingerprinted by its size and its images alone.
+Two witnesses whose fingerprints are equal, page for page, are the same edition of the paper.
+
+How lines are extracted from a PDF is outside this text; the vectors give the lines, so implementations are tested on the function.
+`fingerprints.jsonl` gives two differently stamped downloads of one ACS-style page and of one Wiley-style page that share their fingerprints, a page with one changed digit that does not, and a scan.
+
+## 7.2 Stamp rules
+
+Stamp rules are published in `scienting-stamps/1` files.
+Each rule has an `id`, a `pattern`, a `where` (`any`, or `margin` with a `margin` in points, default 36) and a note.
+A rule matches a line when its pattern is found in the line's normalized text, with ASCII letters compared without case; a `margin` rule matches only lines whose box lies wholly within the margin of one edge.
+Patterns use only syntax that ECMAScript and Python read alike: literals, `.`, `*`, `+`, `?`, `^`, `$`, classes and escaped punctuation.
+
+[registry/stamps/scienting-stamps-1.json](../registry/stamps/scienting-stamps-1.json) holds the published rules.
+A team adds rules it learns, when two witnesses of one version differ only in such lines, in files of its own with `learned: true`.
+A missed stamp splits one edition into two; an over-broad rule hides a real difference.
+
+## 7.3 `pages.jsonl`
+
+| Property | Holds |
+| --- | --- |
+| `witness` | The witness ID. |
+| `page` | The page number in the witness, from 1. |
+| `page_fp` | The fingerprint. |
+| `size` | `[width, height]` in points, after rotation. |
+| `page_label` | The printed page label, when there is one. |
+| `text_layer` | Whether the page has a text layer. |
+| `stamps_removed` | The IDs of the rules that removed lines, in line order, when any did. |
+
+Pages sort by witness ID, then page.
+
+## 7.4 `locations.jsonl`
+
+| Property | Holds |
+| --- | --- |
+| `node` | The node ID. |
+| `page_fp` | The page's fingerprint, which some page of some witness has. |
+| `page` | The page number. |
+| `page_label` | The printed label. |
+| `boxes` | One or more `[x0, y0, x1, y1]` in PDF points, origin at the top left after rotation. |
+| `approx` | `true` when the boxes come from OCR line boxes rather than glyphs. |
+
+A node has one record per page it appears on; a sentence that wraps has several boxes.
+Locations sort by the node's position in document order, then page, then fingerprint.
+Because locations point at fingerprinted pages, a reader can rebuild a document from its own download: it fingerprints the pages, finds the matching ones and reads the text at the stored boxes.