feat: zipped packages, target set, checklists, audit fixes and a stress test
Features for the next six months (accessible HTML export, paperpack and
Lode on packages, the personal bundle):
- Zipped packages: the oracle writes and reads `.sct.zip` archives under
the rules of spec/12 §12.2; validate.py and load.py accept them; a
byte-identical zip of the tiny package joins the fixtures, and two
broken archives (traversal, symbolic link) join fixtures/invalid.
- The target set of targeted F3 (spec/08 §8.10) is computed by the
oracle and pinned by vectors/targets.jsonl under each digit rule; the
rule now covers cells as well as sentences and the leaves of anchored
containers, and defines a node's section.
- scienting-checklist/1 (provisional): a conformance check of an export,
covered by a receipt as ["checklist", id, hash] (spec/08 §8.11), with a
WCAG 2.1 AA fixture and its receipt.
- QA reports are signed like receipts (spec/08 §8.9).
- A `description` annotation body with role alt or long, so an
accessible export can find a confirmed figure description; the layer
fixture carries one, and the store's body-kind constraint allows it.
- scienting-spot-check/1, the seed payload, is registered with a schema.
- exercise, submission and grade are provisional until a course has
used them.
Audit fixes, each with a vector or a broken package where one was
missing:
- Folder packages: symbolic links are refused on validate and load,
manifest paths must not collide under case folding, listed files are
matched by exact name, files are hashed in chunks under a size limit.
- Files are compared byte for byte, so CR LF, a byte order mark, NaN,
a lone surrogate or invalid UTF-8 is refused instead of translated;
integers beyond 2^53 are read as doubles, as canonical JSON writes them.
- Receipts: signature.key must equal by and the key must be valid when
signed; a qa receipt's nodes must be the sample at the original's
hashes, its scope must name qa.of, its errors must be sampled nodes;
statuses are computed once, iteratively, so chains of spot-checks
hundreds deep settle; Base64 and manifest shape errors become refusals
instead of tracebacks; counts and the fidelity of roster-less packages
are recomputed.
- The gate enforces rule 11 of spec/11 §11.4: changed nodes are
validated against the node schema and the references between nodes
are checked, so a set to a bad value, an insert with unknown
properties or a delete of a referenced header cell is refused; a
relabel to null drops the label; a reclassify drops foreign
attributes; a split strips whitespace across runs; ids.jsonl order
follows spec/04 §4.4 in every state; a move after itself is refused.
- Anchors: the relabel rule requires that nothing else changed below the
node; an alias chain ending in a deletion no longer crashes resolve.
- Notation: a script after an empty group applies to the previous atom
({}^{14}C); letters are defined by Unicode block in spec/05 §5.4 and
the oracle; \left) is complex.
- Page fingerprints delete whitespace before folding, so "10 -20" and
"10-20" fold alike (spec/07 §7.1).
- Grades are summed with exact decimals; the reviewer error bound is
computed with integer arithmetic.
- receipts and revocations are keyed by (doc_key, id) in SQL, since one
revocation may be carried by several packages.
Checking:
- tools/stress.py: seeded randomized property tests over canonical JSON,
normalization, notation, random correction walks, anchor resolution,
Merkle roots, spot-check samples, error bounds, fingerprints, mutated
archives and packages, rights and spot-check chains; CI runs it.
- check.py verifies every count and version the prose states against
the repository, the zipped fixture and crafted archives, the checklist
coverage, and nine more broken packages.
- CONTRIBUTING.md gains a release checklist.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
142 files changed
+2312
-268
No files matched your search
+3
-3
@@ -427,8 +427,8 @@ Keep using it as a reading and distribution format.
|
||||
|
||||
*Status.*
|
||||
The specification is version 0.1.0.
|
||||
`registry/formats.json` lists 27 formats: 20 are `candidate` (complete, with vectors and fixtures, ready to implement) and seven are `provisional` (complete; expected to change as implementations gain experience): project, bundle, dataset, digitization, view, snapshot and draft-ids ([spec/00 §0.4](../spec/00-introduction.md#04-status)).
|
||||
The repository holds 23 chapters, 37 JSON Schemas, 20 vector files, five valid fixture packages and 17 invalid ones, each naming the rule it breaks, and a reference checker, `tools/check.py`.
|
||||
`registry/formats.json` lists 29 formats: 18 are `candidate` (complete, with vectors and fixtures, ready to implement) and 11 are `provisional` (complete; expected to change as implementations gain experience): project, bundle, dataset, digitization, view, snapshot, draft-ids, exercise, submission, grade and checklist ([spec/00 §0.4](../spec/00-introduction.md#04-status)).
|
||||
The repository holds 23 chapters, 39 JSON Schemas, 21 vector files, five valid fixture packages, one of them also zipped, and 28 invalid ones, each naming the rule it breaks, and a reference checker, `tools/check.py`.
|
||||
Licenses: prose CC BY 4.0; schemas, SQL, the JSON-LD context, registries, vectors and fixtures CC0 1.0; tools Apache-2.0 ([README.md](../README.md#licenses)).
|
||||
|
||||
*No adoption claim.*
|
||||
@@ -471,7 +471,7 @@ These are the places where the guides and the specification differ, and this pag
|
||||
6. **Package tree:** `docs/concepts.md` omits `suspects.jsonl` from its package tree, and [spec/12 §12.1](../spec/12-package.md#121-files) lists it.
|
||||
7. **RFC 6962 is obsolete:** spec/08 §8.5 cites it; RFC 9162 obsoletes it (see above).
|
||||
8. **Status:** The guides describe the formats as one set.
|
||||
Seven of 27 are `provisional`, including `scienting-dataset/1`, which the datasets paragraphs in `docs/concepts.md` present without that qualifier.
|
||||
Of the 29 formats, 11 are `provisional`, including `scienting-dataset/1`, which the datasets paragraphs in `docs/concepts.md` present without that qualifier.
|
||||
|
||||
*The distinguishing combination, and what it costs.*
|
||||
manuspec puts in one package permanent sentence- and cell-level IDs, five hashes per node, anchors with explicit resolution outcomes and drift, append-only evidence-gated corrections, signed fidelity receipts with spot-check bounds, per-witness rights with AI-use axes, and value-to-cell anchors.
|
||||
|
||||
+3
-2
@@ -12,8 +12,8 @@ For a gentler start, read the [overview](overview.md) first; for the exact rules
|
||||
|
||||
Two facts apply to everything below.
|
||||
The specification is at version 0.1.0.
|
||||
It defines 27 formats, 20 of them `candidate` and 7 `provisional`, as [registry/formats.json](../registry/formats.json) lists.
|
||||
The provisional ones are the project, bundle, dataset, digitization, view, snapshot and draft-ids formats; the sections below say so where they describe them.
|
||||
It defines 29 formats, 18 of them `candidate` and 11 `provisional`, as [registry/formats.json](../registry/formats.json) lists.
|
||||
The provisional ones are the project, bundle, dataset, digitization, view, snapshot, draft-ids, exercise, submission, grade and checklist formats; the sections below say so where they describe them.
|
||||
|
||||
Format identifiers such as `scienting-anchor/1` and `scienting-correction/1` begin with `scienting-`, which is the formats' namespace.
|
||||
The repository's name, manuspec, is a separate matter, and the formats are named after what they hold.
|
||||
@@ -370,6 +370,7 @@ An exercise asks a student to find the cell giving the Set A bond constant for C
|
||||
|
||||
Students submit under pseudonyms, and a key whose anchors have gone stale is never used to grade.
|
||||
Pinned by [vectors/grading.jsonl](../vectors/grading.jsonl).
|
||||
The exercise, submission and grade formats are `provisional`: no course has used them yet, and the first that does will shape them.
|
||||
See [chapter 19](../spec/19-exercises.md).
|
||||
|
||||
## Datasets, digitizations and views
|
||||
|
||||
+3
-3
@@ -96,12 +96,12 @@ The full comparison is [docs/comparison.md](comparison.md)
|
||||
|
||||
## What is its status and maturity?
|
||||
|
||||
Version 0.1.0 has 27 formats: 20 are `candidate` and 7 are `provisional`, as [registry/formats.json](../registry/formats.json) lists.
|
||||
Version 0.1.0 has 29 formats: 18 are `candidate` and 11 are `provisional`, as [registry/formats.json](../registry/formats.json) lists.
|
||||
A `candidate` format is complete, with vectors and fixtures, and ready to implement.
|
||||
A `provisional` one is complete and expected to change as implementations gain experience.
|
||||
The provisional formats are dataset, digitization, view, project, bundle, snapshot and draft-ids.
|
||||
The provisional formats are dataset, digitization, view, project, bundle, snapshot, draft-ids, exercise, submission, grade and checklist.
|
||||
|
||||
The repository holds a reference checker (`tools/check.py`), 20 vector files, and 22 example packages, of which 17 are deliberately broken ([spec/00 §0.4](../spec/00-introduction.md#04-status), [spec/22](../spec/22-conformance.md)).
|
||||
The repository holds a reference checker (`tools/check.py`), 21 vector files, five valid packages, one of them also as a zip, and 28 deliberately broken packages ([spec/00 §0.4](../spec/00-introduction.md#04-status), [spec/22](../spec/22-conformance.md)).
|
||||
|
||||
## What does it not do?
|
||||
|
||||
|
||||
+8
-1
@@ -32,9 +32,16 @@ Format `scienting-bundle/1`, `provisional`. ([13.5](../spec/13-database.md#135-t
|
||||
Changed below
|
||||
: The resolution of an anchor on a container that is unchanged in itself while something inside it changed. ([6.5](../spec/06-anchors.md#65-resolution))
|
||||
|
||||
Checklist
|
||||
: A record of checking an export, such as accessible HTML, against a named standard, criterion by criterion; it is covered by the receipt of the person who checked.
|
||||
Format `scienting-checklist/1`, `provisional`. ([8.11](../spec/08-fidelity.md#811-checklists))
|
||||
|
||||
Correction
|
||||
: An operation on an established document, with its evidence and author; never edited, only appended. ([11](../spec/11-corrections.md))
|
||||
|
||||
Description
|
||||
: An annotation that describes a figure, panel or table for readers who cannot see it, as a short text equivalent (`alt`) or a full description (`long`); an accessible export uses confirmed descriptions only. ([18.4](../spec/18-layers.md#184-bodies))
|
||||
|
||||
Document
|
||||
: The text established for one version of one work, from one or more witnesses, as a tree of nodes. ([2.1](../spec/02-model.md#21-four-objects), [3](../spec/03-nodes.md))
|
||||
|
||||
@@ -135,7 +142,7 @@ Suspect
|
||||
|
||||
Targeted F3
|
||||
: Proofreading the nodes a team's claims, quotations and datasets rest on, rather than every node.
|
||||
The target set also includes every display equation and, under the project's digit rule (`all`, `anchored-sections` or `none`), sentences with a digit in a text or math run (an inline stretch of plain text, or of TeX or Typst math, within the sentence; [3.5](../spec/03-nodes.md#35-inline-runs)). ([8.10](../spec/08-fidelity.md#810-targeted-f3))
|
||||
The target set also includes the leaves of every anchored container, every equation and, under the project's digit rule (`all`, `anchored-sections` or `none`), sentences and cells with a digit in a text or math run (an inline stretch of plain text, or of TeX or Typst math, within the node; [3.5](../spec/03-nodes.md#35-inline-runs)). ([8.10](../spec/08-fidelity.md#810-targeted-f3))
|
||||
|
||||
Text leaf
|
||||
: A leaf whose text a container's projections are made of: any leaf except an equation line, or an equation inside a sentence, whose text is already part of its equation or its sentence. ([3.6](../spec/03-nodes.md#36-the-plain-projection))
|
||||
|
||||
+5
-5
@@ -26,7 +26,7 @@ The formats supply those three, and they are worth most at the scale of one team
|
||||
|
||||
manuspec is a set of file formats for scientific papers that have been converted, split into addressable pieces and checked.
|
||||
The repository is at <https://git.scient.ing/infra/manuspec>.
|
||||
It is version 0.1.0 and defines 27 formats; in [registry/formats.json](../registry/formats.json), 20 are `candidate` and 7 are `provisional`.
|
||||
It is version 0.1.0 and defines 29 formats; in [registry/formats.json](../registry/formats.json), 18 are `candidate` and 11 are `provisional`.
|
||||
|
||||
## Who it is for
|
||||
|
||||
@@ -52,11 +52,11 @@ It appears inside hashed data and test vectors, and it is separate from the repo
|
||||
| Folder | Holds |
|
||||
| --- | --- |
|
||||
| [spec/](../spec/README.md) | The specification, in 23 chapters (00 to 22). Normative. |
|
||||
| [schemas/](../schemas/README.md) | 37 JSON (plain-text data format) Schemas for the files and records. |
|
||||
| [schemas/](../schemas/README.md) | 39 JSON (plain-text data format) Schemas for the files and records. |
|
||||
| [registry/](../registry/README.md) | The list of formats, the rule IDs a checker reports, download-stamp rules, and default rights requirements. |
|
||||
| [tools/](../tools/README.md) | Reference programs: `check.py` checks everything, `build.py` rebuilds the fixtures and vectors, `load.py` loads packages into SQLite, and `oracle/` is a compact implementation of every rule. |
|
||||
| [vectors/](../vectors/README.md) | 20 files of test vectors: inputs and the exact outputs every implementation must compute. |
|
||||
| [fixtures/](../fixtures/README.md) | Five valid example packages, 17 deliberately broken ones that each break one named rule, example records, and the test keys that signed them. All text is synthetic. |
|
||||
| [vectors/](../vectors/README.md) | 21 files of test vectors: inputs and the exact outputs every implementation must compute. |
|
||||
| [fixtures/](../fixtures/README.md) | Five valid example packages, one of them also as a zip, 28 deliberately broken ones that each break one named rule, example records, and the test keys that signed them. All text is synthetic. |
|
||||
| [sql/](../sql/README.md) | Tables for SQLite, Dolt and PostgreSQL, and a SQLite bundle with full-text search. |
|
||||
| [context/](../context/README.md) | The JSON-LD context that makes annotation layers W3C Web Annotations. |
|
||||
| [docs/](README.md) | Guides, including this page. |
|
||||
@@ -107,7 +107,7 @@ The [specification's reading paths](../spec/README.md) say which chapters each r
|
||||
A store loads the package into a database.
|
||||
A reader renders it with layers of annotations.
|
||||
Anchors (stored references to a sentence, with its hash and a quotation) let a claim, a dataset value or a draft's citation point at a passage.
|
||||
Datasets, drafts, views and projects are `provisional` formats.
|
||||
Datasets, drafts, views, projects, exercises and checklists are `provisional` formats.
|
||||
7.
|
||||
Check and sign.
|
||||
People proofread what matters and sign receipts; second reviewers spot-check them.
|
||||
|
||||
+1
-1
@@ -9,7 +9,7 @@ Implementers find the chapter that defines each rule.
|
||||
Experts find the cost of each choice and, where a neighboring standard chose differently, how it differs (the full comparison is in [comparison.md](comparison.md)).
|
||||
|
||||
The formats are at version 0.1.0.
|
||||
Of the 27 in `registry/formats.json`, 20 are `candidate` and 7 are `provisional`, and the provisional ones are named where they appear.
|
||||
Of the 29 in `registry/formats.json`, 18 are `candidate` and 11 are `provisional`, and the provisional ones are named where they appear.
|
||||
The repository is manuspec, at <https://git.scient.ing/infra/manuspec>.
|
||||
The `scienting-` prefix in format names such as `scienting-anchor/1` is the formats' namespace, separate from the repository's name.
|
||||
|
||||
|
||||
+1
-1
@@ -7,7 +7,7 @@ Its problems are modeled on the failures that the specification's rules address.
|
||||
Three readers can use the guide, and each gets something different.
|
||||
A newcomer knows what a paper and a citation are, not hashes or JSON, so every term is defined where it first appears.
|
||||
An implementer who will write a reader, writer, converter or checker gets the chapter and test vector for each step.
|
||||
An expert gets the status: [manuspec](https://git.scient.ing/infra/manuspec) is at version 0.1.0, and each format is `candidate` or `provisional` in [`registry/formats.json`](../registry/formats.json) (20 candidate, 7 provisional).
|
||||
An expert gets the status: [manuspec](https://git.scient.ing/infra/manuspec) is at version 0.1.0, and each format is `candidate` or `provisional` in [`registry/formats.json`](../registry/formats.json) (18 candidate, 11 provisional).
|
||||
The `scienting-` prefix on format names, as in `scienting-doc/1`, is the formats' namespace.
|
||||
It has nothing to do with the repository's name.
|
||||
|
||||
|
||||
Reference in new issue
Block a user