infra/paperspec
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: zipped packages, target set, checklists, audit fixes and a stress test

Browse Source 
Features for the next six months (accessible HTML export, paperpack and
Lode on packages, the personal bundle):

- Zipped packages: the oracle writes and reads `.sct.zip` archives under
  the rules of spec/12 §12.2; validate.py and load.py accept them; a
  byte-identical zip of the tiny package joins the fixtures, and two
  broken archives (traversal, symbolic link) join fixtures/invalid.
- The target set of targeted F3 (spec/08 §8.10) is computed by the
  oracle and pinned by vectors/targets.jsonl under each digit rule; the
  rule now covers cells as well as sentences and the leaves of anchored
  containers, and defines a node's section.
- scienting-checklist/1 (provisional): a conformance check of an export,
  covered by a receipt as ["checklist", id, hash] (spec/08 §8.11), with a
  WCAG 2.1 AA fixture and its receipt.
- QA reports are signed like receipts (spec/08 §8.9).
- A `description` annotation body with role alt or long, so an
  accessible export can find a confirmed figure description; the layer
  fixture carries one, and the store's body-kind constraint allows it.
- scienting-spot-check/1, the seed payload, is registered with a schema.
- exercise, submission and grade are provisional until a course has
  used them.

Audit fixes, each with a vector or a broken package where one was
missing:

- Folder packages: symbolic links are refused on validate and load,
  manifest paths must not collide under case folding, listed files are
  matched by exact name, files are hashed in chunks under a size limit.
- Files are compared byte for byte, so CR LF, a byte order mark, NaN,
  a lone surrogate or invalid UTF-8 is refused instead of translated;
  integers beyond 2^53 are read as doubles, as canonical JSON writes them.
- Receipts: signature.key must equal by and the key must be valid when
  signed; a qa receipt's nodes must be the sample at the original's
  hashes, its scope must name qa.of, its errors must be sampled nodes;
  statuses are computed once, iteratively, so chains of spot-checks
  hundreds deep settle; Base64 and manifest shape errors become refusals
  instead of tracebacks; counts and the fidelity of roster-less packages
  are recomputed.
- The gate enforces rule 11 of spec/11 §11.4: changed nodes are
  validated against the node schema and the references between nodes
  are checked, so a set to a bad value, an insert with unknown
  properties or a delete of a referenced header cell is refused; a
  relabel to null drops the label; a reclassify drops foreign
  attributes; a split strips whitespace across runs; ids.jsonl order
  follows spec/04 §4.4 in every state; a move after itself is refused.
- Anchors: the relabel rule requires that nothing else changed below the
  node; an alias chain ending in a deletion no longer crashes resolve.
- Notation: a script after an empty group applies to the previous atom
  ({}^{14}C); letters are defined by Unicode block in spec/05 §5.4 and
  the oracle; \left) is complex.
- Page fingerprints delete whitespace before folding, so "10 -20" and
  "10-20" fold alike (spec/07 §7.1).
- Grades are summed with exact decimals; the reviewer error bound is
  computed with integer arithmetic.
- receipts and revocations are keyed by (doc_key, id) in SQL, since one
  revocation may be carried by several packages.

Checking:

- tools/stress.py: seeded randomized property tests over canonical JSON,
  normalization, notation, random correction walks, anchor resolution,
  Merkle roots, spot-check samples, error bounds, fingerprints, mutated
  archives and packages, rights and spot-check chains; CI runs it.
- check.py verifies every count and version the prose states against
  the repository, the zipped fixture and crafted archives, the checklist
  coverage, and nine more broken packages.
- CONTRIBUTING.md gains a release checklist.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
alexmandClaude committed 2026-10-14 20:07:22 -04:00
1 parent 8c8b51cc3f
commit ea0a7efa78
142 files changed
+2312 -268
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

schemas/README.md
+7 -7
View File
@@ -1,6 +1,6 @@
# Schemas
This folder holds the JSON (JavaScript Object Notation) Schemas (draft 2020-12) for every file and record that the manuspec formats (manuspec is the specification this repository holds) define: 37 schema files.
This folder holds the JSON (JavaScript Object Notation) Schemas (draft 2020-12) for every file and record that the manuspec formats (manuspec is the specification this repository holds) define: 39 schema files.
A JSON Schema is a file, itself written in JSON, that states what a valid JSON file looks like.
Newcomers can read it as a form: it lists the fields a record must have and what kind of value each field takes.
Implementers will find the loading recipe and the traps under "Validating" and "Writing schemas".
@@ -23,21 +23,21 @@ The number after each name is the chapter of `spec/` that defines the file.
| A package | `manifest.schema.json` (12), `document.schema.json` (2, 12), `node.schema.json` (3), `runs.schema.json` (3), `witnesses.schema.json` (2), `provenance.schema.json` (12), `page.schema.json` (7), `location.schema.json` (7), `reference.schema.json` (10), `id-entry.schema.json` (4), `correction.schema.json` (11), `receipt.schema.json` (8), `revocation.schema.json` (8), `roster.schema.json` (8), `suspect.schema.json` (5), `mathml.schema.json` (12) |
| Pointing into documents | `anchor.schema.json` (6) |
| Rights | `rights.schema.json` (14), `policy.schema.json` (14), `stages.schema.json` (14), `backend.schema.json` (14), `profile.schema.json` (14), `stamps.schema.json` (7) |
| Review | `qa-report.schema.json` (8) |
| Review | `qa-report.schema.json` (8), `spot-check.schema.json` (8), `checklist.schema.json` (8) |
| Annotation and teaching | `layer.schema.json` (18), `annotation.schema.json` (18), `exercise.schema.json` (19), `submission.schema.json` (19), `grade.schema.json` (19) |
| Data | `dataset.schema.json` (15), `value-anchor.schema.json` (15), `digitization.schema.json` (16), `view.schema.json` (17) |
| Teams and drafts | `project.schema.json` (20), `snapshot.schema.json` (21), `draft-ids.schema.json` (21) |
A record that travels on its own carries a `format` such as `scienting-anchor/1`. [registry/formats.json](../registry/formats.json) maps each format to its schema, its chapter and its status.
The formats registry lists 27 formats: 20 are `candidate` (complete, with vectors and fixtures, ready to implement) and 7 are `provisional` (complete; expected to change as implementations gain experience), as defined in `spec/00-introduction.md` §0.4.
The provisional formats are `scienting-project/1`, `scienting-bundle/1`, `scienting-dataset/1`, `scienting-digitization/1`, `scienting-view/1`, `scienting-snapshot/1` and `scienting-draft-ids/1`.
The Data schemas for datasets, digitizations and views, and the Teams and drafts schemas, belong to provisional formats.
The formats registry lists 29 formats: 18 are `candidate` (complete, with vectors and fixtures, ready to implement) and 11 are `provisional` (complete; expected to change as implementations gain experience), as defined in `spec/00-introduction.md` §0.4.
The provisional formats are `scienting-project/1`, `scienting-bundle/1`, `scienting-dataset/1`, `scienting-digitization/1`, `scienting-view/1`, `scienting-snapshot/1`, `scienting-draft-ids/1`, `scienting-exercise/1`, `scienting-submission/1`, `scienting-grade/1` and `scienting-checklist/1`.
The Data schemas for datasets, digitizations and views, the Teams and drafts schemas, the three teaching schemas and `checklist.schema.json` belong to provisional formats.
The rest belong to `candidate` formats or to no format of their own.
The bundle has no JSON Schema: its registry entry points to `sql/bundle-sqlite.sql`.
Eleven schemas have no format of their own because each describes a line or part of a larger file: `common`, `node`, `runs`, `witnesses`, `provenance`, `location`, `reference`, `id-entry`, `mathml`, `annotation` and `value-anchor`.
Of the schemas that do have one, 23 require a `format` property.
Of the 39 schemas, 11 have no format of their own because each describes a line or part of a larger file: `common`, `node`, `runs`, `witnesses`, `provenance`, `location`, `reference`, `id-entry`, `mathml`, `annotation` and `value-anchor`.
Of the schemas that do have one, 25 require a `format` property; `spot-check.schema.json` describes a signing payload rather than a file.
The others (`page`, `draft-ids` and `view`) describe a line of a file or a Vega-Lite specification (a JSON description of a chart), and the record itself does not need to carry the identifier.
## Validating
schemas/annotation.schema.json
+37
View File
@@ -428,6 +428,43 @@
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"type": {
"const": "TextualBody"
},
"value": {
"type": "string",
"minLength": 1
},
"format": {
"enum": [
"text/plain",
"text/markdown"
]
},
"language": {
"$ref": "common.schema.json#/$defs/language"
},
"kind": {
"const": "description"
},
"role": {
"enum": [
"alt",
"long"
]
}
},
"required": [
"type",
"kind",
"value",
"role"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
schemas/checklist.schema.json
+151
View File
@@ -0,0 +1,151 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://scient.ing/ns/schemas/checklist.schema.json",
"title": "Conformance checklist",
"description": "scienting-checklist/1: a check of an export against a named standard, covered by a receipt; spec/08-fidelity.md §8.11.",
"type": "object",
"properties": {
"format": {
"const": "scienting-checklist/1"
},
"id": {
"$ref": "common.schema.json#/$defs/recordId"
},
"doc": {
"$ref": "common.schema.json#/$defs/qualifiedKey"
},
"rev": {
"type": "integer",
"minimum": 0
},
"standard": {
"type": "object",
"properties": {
"name": {
"type": "string",
"minLength": 1
},
"level": {
"type": "string",
"minLength": 1
},
"url": {
"$ref": "common.schema.json#/$defs/url"
}
},
"required": [
"name"
],
"additionalProperties": false
},
"export": {
"type": "object",
"properties": {
"kind": {
"enum": [
"html",
"pdf",
"epub",
"other"
]
},
"sha256": {
"$ref": "common.schema.json#/$defs/hash"
},
"size": {
"type": "integer",
"minimum": 0
}
},
"required": [
"kind",
"sha256"
],
"additionalProperties": false
},
"items": {
"type": "array",
"items": {
"type": "object",
"properties": {
"criterion": {
"type": "string",
"minLength": 1
},
"result": {
"enum": [
"pass",
"fail",
"not-applicable"
]
},
"method": {
"enum": [
"manual",
"automated"
]
},
"tool": {
"type": "object",
"properties": {
"name": {
"type": "string",
"minLength": 1
},
"version": {
"type": "string",
"minLength": 1
}
},
"required": [
"name",
"version"
],
"additionalProperties": false
},
"note": {
"type": "string",
"minLength": 1
}
},
"required": [
"criterion",
"result",
"method"
],
"additionalProperties": false,
"if": {
"properties": {
"method": {
"const": "automated"
}
}
},
"then": {
"required": [
"tool"
]
}
},
"minItems": 1
},
"by": {
"$ref": "common.schema.json#/$defs/key"
},
"at": {
"$ref": "common.schema.json#/$defs/time"
}
},
"required": [
"format",
"id",
"doc",
"rev",
"standard",
"export",
"items",
"by",
"at"
],
"additionalProperties": false
}
schemas/qa-report.schema.json
+9 -1
View File
@@ -67,8 +67,14 @@
},
"minItems": 1
},
"by": {
"$ref": "common.schema.json#/$defs/key"
},
"at": {
"$ref": "common.schema.json#/$defs/time"
},
"signature": {
"$ref": "common.schema.json#/$defs/signature"
}
},
"required": [
@@ -77,7 +83,9 @@
"docs",
"receipts",
"reviewers",
"at"
"by",
"at",
"signature"
],
"additionalProperties": false
}
schemas/spot-check.schema.json
+20
View File
@@ -0,0 +1,20 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://scient.ing/ns/schemas/spot-check.schema.json",
"title": "Spot-check seed payload",
"description": "scienting-spot-check/1: the payload a spot-check key signs to seed a sample; it names a receipt by its record hash and is never a file of its own; spec/08-fidelity.md §8.8.",
"type": "object",
"properties": {
"format": {
"const": "scienting-spot-check/1"
},
"receipt": {
"$ref": "common.schema.json#/$defs/hash"
}
},
"required": [
"format",
"receipt"
],
"additionalProperties": false
}