alexmandClaude
ea0a7efa78
Features for the next six months (accessible HTML export, paperpack and
Lode on packages, the personal bundle):
- Zipped packages: the oracle writes and reads `.sct.zip` archives under
the rules of spec/12 §12.2; validate.py and load.py accept them; a
byte-identical zip of the tiny package joins the fixtures, and two
broken archives (traversal, symbolic link) join fixtures/invalid.
- The target set of targeted F3 (spec/08 §8.10) is computed by the
oracle and pinned by vectors/targets.jsonl under each digit rule; the
rule now covers cells as well as sentences and the leaves of anchored
containers, and defines a node's section.
- scienting-checklist/1 (provisional): a conformance check of an export,
covered by a receipt as ["checklist", id, hash] (spec/08 §8.11), with a
WCAG 2.1 AA fixture and its receipt.
- QA reports are signed like receipts (spec/08 §8.9).
- A `description` annotation body with role alt or long, so an
accessible export can find a confirmed figure description; the layer
fixture carries one, and the store's body-kind constraint allows it.
- scienting-spot-check/1, the seed payload, is registered with a schema.
- exercise, submission and grade are provisional until a course has
used them.
Audit fixes, each with a vector or a broken package where one was
missing:
- Folder packages: symbolic links are refused on validate and load,
manifest paths must not collide under case folding, listed files are
matched by exact name, files are hashed in chunks under a size limit.
- Files are compared byte for byte, so CR LF, a byte order mark, NaN,
a lone surrogate or invalid UTF-8 is refused instead of translated;
integers beyond 2^53 are read as doubles, as canonical JSON writes them.
- Receipts: signature.key must equal by and the key must be valid when
signed; a qa receipt's nodes must be the sample at the original's
hashes, its scope must name qa.of, its errors must be sampled nodes;
statuses are computed once, iteratively, so chains of spot-checks
hundreds deep settle; Base64 and manifest shape errors become refusals
instead of tracebacks; counts and the fidelity of roster-less packages
are recomputed.
- The gate enforces rule 11 of spec/11 §11.4: changed nodes are
validated against the node schema and the references between nodes
are checked, so a set to a bad value, an insert with unknown
properties or a delete of a referenced header cell is refused; a
relabel to null drops the label; a reclassify drops foreign
attributes; a split strips whitespace across runs; ids.jsonl order
follows spec/04 §4.4 in every state; a move after itself is refused.
- Anchors: the relabel rule requires that nothing else changed below the
node; an alias chain ending in a deletion no longer crashes resolve.
- Notation: a script after an empty group applies to the previous atom
({}^{14}C); letters are defined by Unicode block in spec/05 §5.4 and
the oracle; \left) is complex.
- Page fingerprints delete whitespace before folding, so "10 -20" and
"10-20" fold alike (spec/07 §7.1).
- Grades are summed with exact decimals; the reviewer error bound is
computed with integer arithmetic.
- receipts and revocations are keyed by (doc_key, id) in SQL, since one
revocation may be carried by several packages.
Checking:
- tools/stress.py: seeded randomized property tests over canonical JSON,
normalization, notation, random correction walks, anchor resolution,
Merkle roots, spot-check samples, error bounds, fingerprints, mutated
archives and packages, rights and spot-check chains; CI runs it.
- check.py verifies every count and version the prose states against
the repository, the zipped fixture and crafted archives, the checklist
coverage, and nine more broken packages.
- CONTRIBUTING.md gains a release checklist.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
88 lines
7.4 KiB
Markdown
88 lines
7.4 KiB
Markdown
# Schemas
|
|
|
|
This folder holds the JSON (JavaScript Object Notation) Schemas (draft 2020-12) for every file and record that the manuspec formats (manuspec is the specification this repository holds) define: 39 schema files.
|
|
A JSON Schema is a file, itself written in JSON, that states what a valid JSON file looks like.
|
|
Newcomers can read it as a form: it lists the fields a record must have and what kind of value each field takes.
|
|
Implementers will find the loading recipe and the traps under "Validating" and "Writing schemas".
|
|
Experts will find the chapter and status of each format under "Which schema for which file" and "What a schema does not check".
|
|
|
|
Each schema's `$id`, its own address as a name, is `https://scient.ing/ns/schemas/<file>`.
|
|
Schemas refer to each other by those names, and a validator (a program that checks a file against a schema) that loads the whole folder resolves every reference.
|
|
The `$id` base is normative and stays as it is.
|
|
The `scienting-` prefix on format identifiers such as `scienting-anchor/1` is the formats' namespace, a separate matter from the repository's name.
|
|
|
|
## Which schema for which file
|
|
|
|
The schemas are grouped by topic below.
|
|
The number after each name is the chapter of `spec/` that defines the file.
|
|
`common.schema.json` is defined in `spec/01-conventions.md`, the chapter on rules that apply to every format.
|
|
|
|
| Topic | Schemas (chapter) |
|
|
| --- | --- |
|
|
| Shared definitions | `common.schema.json` (1): hashes, IDs, keys, addresses, times, decimals, boxes, licenses, signatures, evidence, quotes. |
|
|
| A package | `manifest.schema.json` (12), `document.schema.json` (2, 12), `node.schema.json` (3), `runs.schema.json` (3), `witnesses.schema.json` (2), `provenance.schema.json` (12), `page.schema.json` (7), `location.schema.json` (7), `reference.schema.json` (10), `id-entry.schema.json` (4), `correction.schema.json` (11), `receipt.schema.json` (8), `revocation.schema.json` (8), `roster.schema.json` (8), `suspect.schema.json` (5), `mathml.schema.json` (12) |
|
|
| Pointing into documents | `anchor.schema.json` (6) |
|
|
| Rights | `rights.schema.json` (14), `policy.schema.json` (14), `stages.schema.json` (14), `backend.schema.json` (14), `profile.schema.json` (14), `stamps.schema.json` (7) |
|
|
| Review | `qa-report.schema.json` (8), `spot-check.schema.json` (8), `checklist.schema.json` (8) |
|
|
| Annotation and teaching | `layer.schema.json` (18), `annotation.schema.json` (18), `exercise.schema.json` (19), `submission.schema.json` (19), `grade.schema.json` (19) |
|
|
| Data | `dataset.schema.json` (15), `value-anchor.schema.json` (15), `digitization.schema.json` (16), `view.schema.json` (17) |
|
|
| Teams and drafts | `project.schema.json` (20), `snapshot.schema.json` (21), `draft-ids.schema.json` (21) |
|
|
|
|
A record that travels on its own carries a `format` such as `scienting-anchor/1`. [registry/formats.json](../registry/formats.json) maps each format to its schema, its chapter and its status.
|
|
|
|
The formats registry lists 29 formats: 18 are `candidate` (complete, with vectors and fixtures, ready to implement) and 11 are `provisional` (complete; expected to change as implementations gain experience), as defined in `spec/00-introduction.md` §0.4.
|
|
The provisional formats are `scienting-project/1`, `scienting-bundle/1`, `scienting-dataset/1`, `scienting-digitization/1`, `scienting-view/1`, `scienting-snapshot/1`, `scienting-draft-ids/1`, `scienting-exercise/1`, `scienting-submission/1`, `scienting-grade/1` and `scienting-checklist/1`.
|
|
The Data schemas for datasets, digitizations and views, the Teams and drafts schemas, the three teaching schemas and `checklist.schema.json` belong to provisional formats.
|
|
The rest belong to `candidate` formats or to no format of their own.
|
|
|
|
The bundle has no JSON Schema: its registry entry points to `sql/bundle-sqlite.sql`.
|
|
Of the 39 schemas, 11 have no format of their own because each describes a line or part of a larger file: `common`, `node`, `runs`, `witnesses`, `provenance`, `location`, `reference`, `id-entry`, `mathml`, `annotation` and `value-anchor`.
|
|
Of the schemas that do have one, 25 require a `format` property; `spot-check.schema.json` describes a signing payload rather than a file.
|
|
The others (`page`, `draft-ids` and `view`) describe a line of a file or a Vega-Lite specification (a JSON description of a chart), and the record itself does not need to carry the identifier.
|
|
|
|
## Validating
|
|
|
|
The example below uses the `jsonschema` package (4.18 or later).
|
|
|
|
```python
|
|
import json, pathlib
|
|
from jsonschema import Draft202012Validator
|
|
from referencing import Registry, Resource
|
|
|
|
schemas = {p.name: json.loads(p.read_text()) for p in pathlib.Path("schemas").glob("*.json")}
|
|
registry = Registry().with_resources((s["$id"], Resource.from_contents(s)) for s in schemas.values())
|
|
validator = Draft202012Validator(schemas["anchor.schema.json"], registry=registry)
|
|
validator.validate(json.load(open("fixtures/records/anchor-span.json")))
|
|
```
|
|
|
|
It raises no error when run from the repository root, since `anchor-span.json` is a valid anchor.
|
|
A schema refers to another by a relative name, and the `Registry` object resolves it.
|
|
Here `anchor.schema.json` points to `common.schema.json#/$defs/key`, which a validator resolves against the `$id` above into `https://scient.ing/ns/schemas/common.schema.json`.
|
|
The `Registry` maps that name to the file's contents, so nothing is fetched over the network.
|
|
`Draft202012Validator(schemas["anchor.schema.json"])` fails with `Unresolvable: common.schema.json#/$defs/key` when the `Registry` is missing.
|
|
Never load a schema alone.
|
|
|
|
Any draft 2020-12 validator works the same way: give it every schema under its `$id`, then validate against the one you need.
|
|
`python3 tools/check.py` validates every fixture with `jsonschema` this way, and checks that every schema keeps to the keyword subset.
|
|
|
|
## What a schema does not check
|
|
|
|
A schema checks shape: required properties, types, patterns, enumerations.
|
|
The anchor in `fixtures/records/anchor-span.json` has a `node_hash` (a fingerprint of the node's content) of 64 lowercase hex digits, and a schema can confirm exactly that.
|
|
It cannot confirm that the hash is the right one: replace it with 64 zeros and the record still validates.
|
|
It also cannot check that a node's children are allowed or that a receipt's signature verifies, nor that a correction's ID derives from its content.
|
|
The rules of [spec/22-conformance.md](../spec/22-conformance.md) cover these checks, each with an ID in [registry/rules.json](../registry/rules.json).
|
|
For the four examples the IDs are `doc.hash.node-hash`, `doc.tree.child`, `doc.correction.id` and `doc.receipt.signature`.
|
|
|
|
A validator that reports `package.schema` has found only a shape error.
|
|
The other rules need the chapters' computations, and the test vectors in `vectors/` (for example `hashes.jsonl`, `corrections.jsonl` and `signatures.jsonl`) give the expected outputs.
|
|
The schemas are normative, and conformance has further steps: spec/22 §22.2 lists them in order.
|
|
|
|
## Writing schemas
|
|
|
|
Validators in many languages read the same schemas, which limits how the schemas are written.
|
|
The schemas use only the keywords listed in [spec/22-conformance.md](../spec/22-conformance.md#223-schemas).
|
|
Their patterns avoid `\d`, `\w`, `\s`, `\b` and lookbehind, so validators in every language read them alike.
|
|
The repository's own validator refuses a schema that uses any other keyword or those pattern tokens.
|
|
Optional properties are omitted rather than set to null, except where a schema says a property is nullable.
|