alexmandClaude
ea0a7efa78
Features for the next six months (accessible HTML export, paperpack and
Lode on packages, the personal bundle):
- Zipped packages: the oracle writes and reads `.sct.zip` archives under
the rules of spec/12 §12.2; validate.py and load.py accept them; a
byte-identical zip of the tiny package joins the fixtures, and two
broken archives (traversal, symbolic link) join fixtures/invalid.
- The target set of targeted F3 (spec/08 §8.10) is computed by the
oracle and pinned by vectors/targets.jsonl under each digit rule; the
rule now covers cells as well as sentences and the leaves of anchored
containers, and defines a node's section.
- scienting-checklist/1 (provisional): a conformance check of an export,
covered by a receipt as ["checklist", id, hash] (spec/08 §8.11), with a
WCAG 2.1 AA fixture and its receipt.
- QA reports are signed like receipts (spec/08 §8.9).
- A `description` annotation body with role alt or long, so an
accessible export can find a confirmed figure description; the layer
fixture carries one, and the store's body-kind constraint allows it.
- scienting-spot-check/1, the seed payload, is registered with a schema.
- exercise, submission and grade are provisional until a course has
used them.
Audit fixes, each with a vector or a broken package where one was
missing:
- Folder packages: symbolic links are refused on validate and load,
manifest paths must not collide under case folding, listed files are
matched by exact name, files are hashed in chunks under a size limit.
- Files are compared byte for byte, so CR LF, a byte order mark, NaN,
a lone surrogate or invalid UTF-8 is refused instead of translated;
integers beyond 2^53 are read as doubles, as canonical JSON writes them.
- Receipts: signature.key must equal by and the key must be valid when
signed; a qa receipt's nodes must be the sample at the original's
hashes, its scope must name qa.of, its errors must be sampled nodes;
statuses are computed once, iteratively, so chains of spot-checks
hundreds deep settle; Base64 and manifest shape errors become refusals
instead of tracebacks; counts and the fidelity of roster-less packages
are recomputed.
- The gate enforces rule 11 of spec/11 §11.4: changed nodes are
validated against the node schema and the references between nodes
are checked, so a set to a bad value, an insert with unknown
properties or a delete of a referenced header cell is refused; a
relabel to null drops the label; a reclassify drops foreign
attributes; a split strips whitespace across runs; ids.jsonl order
follows spec/04 §4.4 in every state; a move after itself is refused.
- Anchors: the relabel rule requires that nothing else changed below the
node; an alias chain ending in a deletion no longer crashes resolve.
- Notation: a script after an empty group applies to the previous atom
({}^{14}C); letters are defined by Unicode block in spec/05 §5.4 and
the oracle; \left) is complex.
- Page fingerprints delete whitespace before folding, so "10 -20" and
"10-20" fold alike (spec/07 §7.1).
- Grades are summed with exact decimals; the reviewer error bound is
computed with integer arithmetic.
- receipts and revocations are keyed by (doc_key, id) in SQL, since one
revocation may be carried by several packages.
Checking:
- tools/stress.py: seeded randomized property tests over canonical JSON,
normalization, notation, random correction walks, anchor resolution,
Merkle roots, spot-check samples, error bounds, fingerprints, mutated
archives and packages, rights and spot-check chains; CI runs it.
- check.py verifies every count and version the prose states against
the repository, the zipped fixture and crafted archives, the checklist
coverage, and nine more broken packages.
- CONTRIBUTING.md gains a release checklist.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
80 lines
5.2 KiB
Markdown
80 lines
5.2 KiB
Markdown
# Registry
|
|
|
|
The registry holds lists that tools read at run time.
|
|
Each list is a data file, so a reader, writer, converter or checker loads it instead of copying it into code, and a team can change a list without changing a program.
|
|
This page is for newcomers, implementers who need the exact behavior, and experts who want the status of each format.
|
|
|
|
| File | Holds |
|
|
| --- | --- |
|
|
| `formats.json` | Every format: its identifier, media type (null for the five that have none), schema, status, chapter and a one-line description. |
|
|
| `rules.json` | Every validation rule ID, with what it means and the chapter that defines it. |
|
|
| `stamps/scienting-stamps-1.json` | The published rules that recognize the download stamps publishers add to PDFs (see Adding stamp rules), so page fingerprints ignore them. |
|
|
| `stages/scienting-stages-default.json` | The rights each processing stage needs, as a default a team may replace. |
|
|
|
|
## Format status
|
|
|
|
Each entry in `formats.json` has a `status`.
|
|
A `candidate` format is complete, with vectors and fixtures, and ready to implement.
|
|
It changes only with a recorded reason and a new or changed vector.
|
|
A `provisional` format is complete and expected to change as implementations gain experience, so it may change more freely.
|
|
At version 0.1.0, 18 of the 29 formats are `candidate` and 11 are `provisional`.
|
|
The third defined value, `reserved`, is unused.
|
|
The values are defined in [Chapter 0, section 0.4](../spec/00-introduction.md#04-status), and the stability rules in [Chapter 22, section 22.5](../spec/22-conformance.md#225-stability).
|
|
|
|
## Rule IDs
|
|
|
|
A validator checks a package and reports each failure as `path:line: rule: message`, for example:
|
|
|
|
```text
|
|
nodes.jsonl:6: doc.hash.node-hash: node_hash does not recompute
|
|
```
|
|
|
|
A `node_hash` is a hash (a fixed-length value computed from content, so any change to the content changes it) stored with each node of a document.
|
|
The line above is the failure that `fixtures/invalid/hash-mismatch/` is built to cause: its sixth node record carries a `node_hash` of all zeros.
|
|
|
|
A rule's meaning never changes, and a retired rule's ID is never reused, so tools and people can rely on the IDs.
|
|
A tool can match on the `rule` field, whose meaning is looked up in `rules.json`, and treat the message as text for people.
|
|
[Chapter 22, section 22.4](../spec/22-conformance.md#224-rules) lists the ID prefixes.
|
|
Each directory in `fixtures/invalid/` has an `expect.json` that names the rule it breaks.
|
|
The reference checker (`tools/check.py`) fails if that rule is not in `rules.json`.
|
|
|
|
## Adding stamp rules
|
|
|
|
Many publishers add a line to each PDF they serve, naming the institution and the time of download.
|
|
This page calls that line a stamp.
|
|
Two downloads of one paper then differ, so a hash of the file cannot match them.
|
|
A page fingerprint is a hash of a page's size, its text and its images.
|
|
It is computed after removing every line that a stamp rule matches ([Chapter 7, section 7.1](../spec/07-geometry.md#71-page-fingerprints)).
|
|
The vectors in `vectors/fingerprints.jsonl` show this: `acs-download-1` and `acs-download-2` differ in the institution and the time on their stamp lines and have the same fingerprint.
|
|
|
|
Suppose two downloads of one paper differ only in a stamp line that no published rule recognizes.
|
|
Record a rule in a file of your own, marked as learned:
|
|
|
|
```json
|
|
{"format": "scienting-stamps/1", "id": "smithlab-learned", "learned": true,
|
|
"rules": [{"id": "licensed-to", "pattern": "^This copy is licensed to .+", "where": "margin", "margin": 36}]}
|
|
```
|
|
|
|
Normalized text is the line after Unicode Normalization Form C, with each run of whitespace replaced by one space and the ends trimmed ([Chapter 1, section 1.4](../spec/01-conventions.md#14-text-normalization-for-hashing)).
|
|
|
|
A rule has an `id`, a `pattern`, and a `where` that is `any` or `margin`.
|
|
A `margin` rule matches only lines whose box lies wholly within `margin` points of one page edge (36 when the rule gives none).
|
|
Patterns use only syntax that ECMAScript and Python read alike, and match a line's normalized text, with the ASCII letters A to Z compared without case.
|
|
The allowed syntax is literals, `.`, `*`, `+`, `?`, `^`, `$`, classes and escaped punctuation.
|
|
|
|
A page record in `pages.jsonl` lists, in `stamps_removed`, the IDs of the rules that removed lines.
|
|
A rule that is too broad hides a real difference between two pages.
|
|
A stamp that no rule recognizes splits one edition into two ([Chapter 7, section 7.2](../spec/07-geometry.md#72-stamp-rules)).
|
|
|
|
## Replacing the stage defaults
|
|
|
|
A stage is one step of processing, such as conversion or indexing.
|
|
Each stage needs certain rights, such as the right to mine text, and a stage runs only when the rights record of the paper allows them.
|
|
The stage requirements are configuration, not code, because what a team may do depends on how it holds each paper.
|
|
|
|
A team copies `scienting-stages-default.json`, adjusts it with its counsel and points its project's settings at the copy.
|
|
The file itself calls its contents "proposed defaults from the design, for counsel to set".
|
|
In a project, the `settings.stages` field holds the `id` of the stage file in use ([Chapter 20](../spec/20-projects.md)), so a team's copy needs an `id` of its own.
|
|
The project format is `provisional`.
|
|
[Chapter 14](../spec/14-rights.md) explains how stages are checked.
|