feat: zipped packages, target set, checklists, audit fixes and a stress test
Features for the next six months (accessible HTML export, paperpack and
Lode on packages, the personal bundle):
- Zipped packages: the oracle writes and reads `.sct.zip` archives under
the rules of spec/12 §12.2; validate.py and load.py accept them; a
byte-identical zip of the tiny package joins the fixtures, and two
broken archives (traversal, symbolic link) join fixtures/invalid.
- The target set of targeted F3 (spec/08 §8.10) is computed by the
oracle and pinned by vectors/targets.jsonl under each digit rule; the
rule now covers cells as well as sentences and the leaves of anchored
containers, and defines a node's section.
- scienting-checklist/1 (provisional): a conformance check of an export,
covered by a receipt as ["checklist", id, hash] (spec/08 §8.11), with a
WCAG 2.1 AA fixture and its receipt.
- QA reports are signed like receipts (spec/08 §8.9).
- A `description` annotation body with role alt or long, so an
accessible export can find a confirmed figure description; the layer
fixture carries one, and the store's body-kind constraint allows it.
- scienting-spot-check/1, the seed payload, is registered with a schema.
- exercise, submission and grade are provisional until a course has
used them.
Audit fixes, each with a vector or a broken package where one was
missing:
- Folder packages: symbolic links are refused on validate and load,
manifest paths must not collide under case folding, listed files are
matched by exact name, files are hashed in chunks under a size limit.
- Files are compared byte for byte, so CR LF, a byte order mark, NaN,
a lone surrogate or invalid UTF-8 is refused instead of translated;
integers beyond 2^53 are read as doubles, as canonical JSON writes them.
- Receipts: signature.key must equal by and the key must be valid when
signed; a qa receipt's nodes must be the sample at the original's
hashes, its scope must name qa.of, its errors must be sampled nodes;
statuses are computed once, iteratively, so chains of spot-checks
hundreds deep settle; Base64 and manifest shape errors become refusals
instead of tracebacks; counts and the fidelity of roster-less packages
are recomputed.
- The gate enforces rule 11 of spec/11 §11.4: changed nodes are
validated against the node schema and the references between nodes
are checked, so a set to a bad value, an insert with unknown
properties or a delete of a referenced header cell is refused; a
relabel to null drops the label; a reclassify drops foreign
attributes; a split strips whitespace across runs; ids.jsonl order
follows spec/04 §4.4 in every state; a move after itself is refused.
- Anchors: the relabel rule requires that nothing else changed below the
node; an alias chain ending in a deletion no longer crashes resolve.
- Notation: a script after an empty group applies to the previous atom
({}^{14}C); letters are defined by Unicode block in spec/05 §5.4 and
the oracle; \left) is complex.
- Page fingerprints delete whitespace before folding, so "10 -20" and
"10-20" fold alike (spec/07 §7.1).
- Grades are summed with exact decimals; the reviewer error bound is
computed with integer arithmetic.
- receipts and revocations are keyed by (doc_key, id) in SQL, since one
revocation may be carried by several packages.
Checking:
- tools/stress.py: seeded randomized property tests over canonical JSON,
normalization, notation, random correction walks, anchor resolution,
Merkle roots, spot-check samples, error bounds, fingerprints, mutated
archives and packages, rights and spot-check chains; CI runs it.
- check.py verifies every count and version the prose states against
the repository, the zipped fixture and crafted archives, the checklist
coverage, and nine more broken packages.
- CONTRIBUTING.md gains a release checklist.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
142 files changed
+2312
-268
No files matched your search
+5
-5
@@ -26,7 +26,7 @@ The formats supply those three, and they are worth most at the scale of one team
|
||||
|
||||
manuspec is a set of file formats for scientific papers that have been converted, split into addressable pieces and checked.
|
||||
The repository is at <https://git.scient.ing/infra/manuspec>.
|
||||
It is version 0.1.0 and defines 27 formats; in [registry/formats.json](../registry/formats.json), 20 are `candidate` and 7 are `provisional`.
|
||||
It is version 0.1.0 and defines 29 formats; in [registry/formats.json](../registry/formats.json), 18 are `candidate` and 11 are `provisional`.
|
||||
|
||||
## Who it is for
|
||||
|
||||
@@ -52,11 +52,11 @@ It appears inside hashed data and test vectors, and it is separate from the repo
|
||||
| Folder | Holds |
|
||||
| --- | --- |
|
||||
| [spec/](../spec/README.md) | The specification, in 23 chapters (00 to 22). Normative. |
|
||||
| [schemas/](../schemas/README.md) | 37 JSON (plain-text data format) Schemas for the files and records. |
|
||||
| [schemas/](../schemas/README.md) | 39 JSON (plain-text data format) Schemas for the files and records. |
|
||||
| [registry/](../registry/README.md) | The list of formats, the rule IDs a checker reports, download-stamp rules, and default rights requirements. |
|
||||
| [tools/](../tools/README.md) | Reference programs: `check.py` checks everything, `build.py` rebuilds the fixtures and vectors, `load.py` loads packages into SQLite, and `oracle/` is a compact implementation of every rule. |
|
||||
| [vectors/](../vectors/README.md) | 20 files of test vectors: inputs and the exact outputs every implementation must compute. |
|
||||
| [fixtures/](../fixtures/README.md) | Five valid example packages, 17 deliberately broken ones that each break one named rule, example records, and the test keys that signed them. All text is synthetic. |
|
||||
| [vectors/](../vectors/README.md) | 21 files of test vectors: inputs and the exact outputs every implementation must compute. |
|
||||
| [fixtures/](../fixtures/README.md) | Five valid example packages, one of them also as a zip, 28 deliberately broken ones that each break one named rule, example records, and the test keys that signed them. All text is synthetic. |
|
||||
| [sql/](../sql/README.md) | Tables for SQLite, Dolt and PostgreSQL, and a SQLite bundle with full-text search. |
|
||||
| [context/](../context/README.md) | The JSON-LD context that makes annotation layers W3C Web Annotations. |
|
||||
| [docs/](README.md) | Guides, including this page. |
|
||||
@@ -107,7 +107,7 @@ The [specification's reading paths](../spec/README.md) say which chapters each r
|
||||
A store loads the package into a database.
|
||||
A reader renders it with layers of annotations.
|
||||
Anchors (stored references to a sentence, with its hash and a quotation) let a claim, a dataset value or a draft's citation point at a passage.
|
||||
Datasets, drafts, views and projects are `provisional` formats.
|
||||
Datasets, drafts, views, projects, exercises and checklists are `provisional` formats.
|
||||
7.
|
||||
Check and sign.
|
||||
People proofread what matters and sign receipts; second reviewers spot-check them.
|
||||
|
||||
Reference in new issue
Block a user