alexmandClaude
ea0a7efa78
Features for the next six months (accessible HTML export, paperpack and
Lode on packages, the personal bundle):
- Zipped packages: the oracle writes and reads `.sct.zip` archives under
the rules of spec/12 §12.2; validate.py and load.py accept them; a
byte-identical zip of the tiny package joins the fixtures, and two
broken archives (traversal, symbolic link) join fixtures/invalid.
- The target set of targeted F3 (spec/08 §8.10) is computed by the
oracle and pinned by vectors/targets.jsonl under each digit rule; the
rule now covers cells as well as sentences and the leaves of anchored
containers, and defines a node's section.
- scienting-checklist/1 (provisional): a conformance check of an export,
covered by a receipt as ["checklist", id, hash] (spec/08 §8.11), with a
WCAG 2.1 AA fixture and its receipt.
- QA reports are signed like receipts (spec/08 §8.9).
- A `description` annotation body with role alt or long, so an
accessible export can find a confirmed figure description; the layer
fixture carries one, and the store's body-kind constraint allows it.
- scienting-spot-check/1, the seed payload, is registered with a schema.
- exercise, submission and grade are provisional until a course has
used them.
Audit fixes, each with a vector or a broken package where one was
missing:
- Folder packages: symbolic links are refused on validate and load,
manifest paths must not collide under case folding, listed files are
matched by exact name, files are hashed in chunks under a size limit.
- Files are compared byte for byte, so CR LF, a byte order mark, NaN,
a lone surrogate or invalid UTF-8 is refused instead of translated;
integers beyond 2^53 are read as doubles, as canonical JSON writes them.
- Receipts: signature.key must equal by and the key must be valid when
signed; a qa receipt's nodes must be the sample at the original's
hashes, its scope must name qa.of, its errors must be sampled nodes;
statuses are computed once, iteratively, so chains of spot-checks
hundreds deep settle; Base64 and manifest shape errors become refusals
instead of tracebacks; counts and the fidelity of roster-less packages
are recomputed.
- The gate enforces rule 11 of spec/11 §11.4: changed nodes are
validated against the node schema and the references between nodes
are checked, so a set to a bad value, an insert with unknown
properties or a delete of a referenced header cell is refused; a
relabel to null drops the label; a reclassify drops foreign
attributes; a split strips whitespace across runs; ids.jsonl order
follows spec/04 §4.4 in every state; a move after itself is refused.
- Anchors: the relabel rule requires that nothing else changed below the
node; an alias chain ending in a deletion no longer crashes resolve.
- Notation: a script after an empty group applies to the previous atom
({}^{14}C); letters are defined by Unicode block in spec/05 §5.4 and
the oracle; \left) is complex.
- Page fingerprints delete whitespace before folding, so "10 -20" and
"10-20" fold alike (spec/07 §7.1).
- Grades are summed with exact decimals; the reviewer error bound is
computed with integer arithmetic.
- receipts and revocations are keyed by (doc_key, id) in SQL, since one
revocation may be carried by several packages.
Checking:
- tools/stress.py: seeded randomized property tests over canonical JSON,
normalization, notation, random correction walks, anchor resolution,
Merkle roots, spot-check samples, error bounds, fingerprints, mutated
archives and packages, rights and spot-check chains; CI runs it.
- check.py verifies every count and version the prose states against
the repository, the zipped fixture and crafted archives, the checklist
coverage, and nine more broken packages.
- CONTRIBUTING.md gains a release checklist.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Registry
The registry holds lists that tools read at run time. Each list is a data file, so a reader, writer, converter or checker loads it instead of copying it into code, and a team can change a list without changing a program. This page is for newcomers, implementers who need the exact behavior, and experts who want the status of each format.
| File | Holds |
|---|---|
formats.json |
Every format: its identifier, media type (null for the five that have none), schema, status, chapter and a one-line description. |
rules.json |
Every validation rule ID, with what it means and the chapter that defines it. |
stamps/scienting-stamps-1.json |
The published rules that recognize the download stamps publishers add to PDFs (see Adding stamp rules), so page fingerprints ignore them. |
stages/scienting-stages-default.json |
The rights each processing stage needs, as a default a team may replace. |
Format status
Each entry in formats.json has a status.
A candidate format is complete, with vectors and fixtures, and ready to implement.
It changes only with a recorded reason and a new or changed vector.
A provisional format is complete and expected to change as implementations gain experience, so it may change more freely.
At version 0.1.0, 18 of the 29 formats are candidate and 11 are provisional.
The third defined value, reserved, is unused.
The values are defined in Chapter 0, section 0.4, and the stability rules in Chapter 22, section 22.5.
Rule IDs
A validator checks a package and reports each failure as path:line: rule: message, for example:
nodes.jsonl:6: doc.hash.node-hash: node_hash does not recompute
A node_hash is a hash (a fixed-length value computed from content, so any change to the content changes it) stored with each node of a document.
The line above is the failure that fixtures/invalid/hash-mismatch/ is built to cause: its sixth node record carries a node_hash of all zeros.
A rule's meaning never changes, and a retired rule's ID is never reused, so tools and people can rely on the IDs.
A tool can match on the rule field, whose meaning is looked up in rules.json, and treat the message as text for people.
Chapter 22, section 22.4 lists the ID prefixes.
Each directory in fixtures/invalid/ has an expect.json that names the rule it breaks.
The reference checker (tools/check.py) fails if that rule is not in rules.json.
Adding stamp rules
Many publishers add a line to each PDF they serve, naming the institution and the time of download.
This page calls that line a stamp.
Two downloads of one paper then differ, so a hash of the file cannot match them.
A page fingerprint is a hash of a page's size, its text and its images.
It is computed after removing every line that a stamp rule matches (Chapter 7, section 7.1).
The vectors in vectors/fingerprints.jsonl show this: acs-download-1 and acs-download-2 differ in the institution and the time on their stamp lines and have the same fingerprint.
Suppose two downloads of one paper differ only in a stamp line that no published rule recognizes. Record a rule in a file of your own, marked as learned:
{"format": "scienting-stamps/1", "id": "smithlab-learned", "learned": true,
"rules": [{"id": "licensed-to", "pattern": "^This copy is licensed to .+", "where": "margin", "margin": 36}]}
Normalized text is the line after Unicode Normalization Form C, with each run of whitespace replaced by one space and the ends trimmed (Chapter 1, section 1.4).
A rule has an id, a pattern, and a where that is any or margin.
A margin rule matches only lines whose box lies wholly within margin points of one page edge (36 when the rule gives none).
Patterns use only syntax that ECMAScript and Python read alike, and match a line's normalized text, with the ASCII letters A to Z compared without case.
The allowed syntax is literals, ., *, +, ?, ^, $, classes and escaped punctuation.
A page record in pages.jsonl lists, in stamps_removed, the IDs of the rules that removed lines.
A rule that is too broad hides a real difference between two pages.
A stamp that no rule recognizes splits one edition into two (Chapter 7, section 7.2).
Replacing the stage defaults
A stage is one step of processing, such as conversion or indexing. Each stage needs certain rights, such as the right to mine text, and a stage runs only when the rights record of the paper allows them. The stage requirements are configuration, not code, because what a team may do depends on how it holds each paper.
A team copies scienting-stages-default.json, adjusts it with its counsel and points its project's settings at the copy.
The file itself calls its contents "proposed defaults from the design, for counsel to set".
In a project, the settings.stages field holds the id of the stage file in use (Chapter 20), so a team's copy needs an id of its own.
The project format is provisional.
Chapter 14 explains how stages are checked.