paperspec/schemas/README.md

# Schemas

This folder holds the JSON (JavaScript Object Notation) Schemas (draft 2020-12) for every file and record that the paperspec formats (paperspec is the specification this repository holds) define: 42 schema files.
A JSON Schema is a file, itself written in JSON, that states what a valid JSON file looks like.
Newcomers can read it as a form: it lists the fields a record must have and what kind of value each field takes.
Implementers will find the loading recipe and the traps under "Validating" and "Writing schemas".
Experts will find the chapter and status of each format under "Which schema for which file" and "What a schema does not check".

Each schema's `$id`, its own address as a name, is `https://paperspec.scient.ing/schemas/<file>`.
Schemas refer to each other by those names, and a validator (a program that checks a file against a schema) that loads the whole folder resolves every reference.
The `$id` base is normative and stays as it is.
The `scienting-` prefix on format identifiers such as `scienting-anchor/1` is the formats' namespace, a separate matter from the repository's name.

## Which schema for which file

The schemas are grouped by topic below.
The number after each name is the chapter of `spec/` that defines the file.
`common.schema.json` is defined in `spec/01-conventions.md`, the chapter on rules that apply to every format.

| Topic | Schemas (chapter) |
| --- | --- |
| Shared definitions | `common.schema.json` (1): hashes, IDs, keys, addresses, times, decimals, boxes, licenses, signatures, evidence, quotes. |
| A package | `manifest.schema.json` (12), `document.schema.json` (2, 12), `node.schema.json` (3), `runs.schema.json` (3), `witnesses.schema.json` (2), `provenance.schema.json` (12), `page.schema.json` (7), `location.schema.json` (7), `reference.schema.json` (10), `id-entry.schema.json` (4), `correction.schema.json` (11), `receipt.schema.json` (8), `revocation.schema.json` (8), `roster.schema.json` (8), `suspect.schema.json` (5), `mathml.schema.json` (12) |
| Pointing into documents | `anchor.schema.json` (6) |
| Rights | `rights.schema.json` (14), `policy.schema.json` (14), `stages.schema.json` (14), `backend.schema.json` (14), `profile.schema.json` (14), `stamps.schema.json` (7) |
| Review | `qa-report.schema.json` (8), `spot-check.schema.json` (8), `checklist.schema.json` (8), `proof.schema.json` (8, 11) |
| The transparency log | `log-entry.schema.json` (23), `checkpoint.schema.json` (23) |
| Annotation and teaching | `layer.schema.json` (18), `annotation.schema.json` (18), `exercise.schema.json` (19), `submission.schema.json` (19), `grade.schema.json` (19) |
| Data | `dataset.schema.json` (15), `value-anchor.schema.json` (15), `digitization.schema.json` (16), `view.schema.json` (17) |
| Teams and drafts | `project.schema.json` (20), `snapshot.schema.json` (21), `draft-ids.schema.json` (21) |

A record that travels on its own carries a `format` such as `scienting-anchor/1`. [registry/formats.json](../registry/formats.json) maps each format to its schema, its chapter and its status.

The formats registry lists 32 formats: 19 are `candidate` (complete, with vectors and fixtures, ready to implement) and 13 are `provisional` (complete; expected to change as implementations gain experience), as defined in `spec/00-introduction.md` §0.4.
The provisional formats are `scienting-project/1`, `scienting-bundle/1`, `scienting-dataset/1`, `scienting-digitization/1`, `scienting-view/1`, `scienting-snapshot/1`, `scienting-draft-ids/1`, `scienting-exercise/1`, `scienting-submission/1`, `scienting-grade/1`, `scienting-checklist/1`, `scienting-log-entry/1` and `scienting-checkpoint/1`.
The Data schemas for datasets, digitizations and views, the Teams and drafts schemas, the three teaching schemas, `checklist.schema.json` and the two log schemas belong to provisional formats.
The rest belong to `candidate` formats or to no format of their own.

The bundle has no JSON Schema: its registry entry points to `sql/bundle-sqlite.sql`.
Of the 42 schemas, 11 have no format of their own because each describes a line or part of a larger file: `common`, `node`, `runs`, `witnesses`, `provenance`, `location`, `reference`, `id-entry`, `mathml`, `annotation` and `value-anchor`.
Of the schemas that do have one, 27 require a `format` property; `spot-check.schema.json` describes the input a VRF proof is made over rather than a file.
The others (`page`, `draft-ids`, `view` and `log-entry`) describe a line of a file, a Vega-Lite specification (a JSON description of a chart) or an item of a tree, and the record itself does not need to carry the identifier.

## Validating

The example below uses the `jsonschema` package (4.18 or later).

```python
import json, pathlib
from jsonschema import Draft202012Validator
from referencing import Registry, Resource

schemas = {p.name: json.loads(p.read_text()) for p in pathlib.Path("schemas").glob("*.json")}
registry = Registry().with_resources((s["$id"], Resource.from_contents(s)) for s in schemas.values())
validator = Draft202012Validator(schemas["anchor.schema.json"], registry=registry)
validator.validate(json.load(open("fixtures/records/anchor-span.json")))
```

It raises no error when run from the repository root, since `anchor-span.json` is a valid anchor.
A schema refers to another by a relative name, and the `Registry` object resolves it.
Here `anchor.schema.json` points to `common.schema.json#/$defs/key`, which a validator resolves against the `$id` above into `https://paperspec.scient.ing/schemas/common.schema.json`.
The `Registry` maps that name to the file's contents, so nothing is fetched over the network.
`Draft202012Validator(schemas["anchor.schema.json"])` fails with `Unresolvable: common.schema.json#/$defs/key` when the `Registry` is missing.
Never load a schema alone.

Any draft 2020-12 validator works the same way: give it every schema under its `$id`, then validate against the one you need.
`paperspec check` validates every fixture with `jsonschema` this way, and checks that every schema keeps to the keyword subset.

## What a schema does not check

A schema checks shape: required properties, types, patterns, enumerations.
The anchor in `fixtures/records/anchor-span.json` has a `node_hash` (a fingerprint of the node's content) of 64 lowercase hex digits, and a schema can confirm exactly that.
It cannot confirm that the hash is the right one: replace it with 64 zeros and the record still validates.
It also cannot check that a node's children are allowed or that a receipt's signature verifies, nor that a correction's ID derives from its content.
The rules of [spec/22-conformance.md](../spec/22-conformance.md) cover these checks, each with an ID in [registry/rules.json](../registry/rules.json).
For the four examples the IDs are `doc.hash.node-hash`, `doc.tree.child`, `doc.correction.id` and `doc.receipt.signature`.

A validator that reports `package.schema` has found only a shape error.
The other rules need the chapters' computations, and the test vectors in `vectors/` (for example `hashes.jsonl`, `corrections.jsonl` and `signatures.jsonl`) give the expected outputs.
The schemas are normative, and conformance has further steps: spec/22 §22.2 lists them in order.

## Writing schemas

Validators in many languages read the same schemas, which limits how the schemas are written.
The schemas use only the keywords listed in [spec/22-conformance.md](../spec/22-conformance.md#223-schemas).
Their patterns avoid `\d`, `\w`, `\s`, `\b` and lookbehind, so validators in every language read them alike.
The repository's own validator refuses a schema that uses any other keyword or those pattern tokens.
Optional properties are omitted rather than set to null, except where a schema says a property is nullable.