Files
paperspec/spec/15-datasets.md
T
alexmandClaude 4fb8b4f5a7 Add paperspec correct, check math forms and datasets, and give the tools public names
The work the other paper tools planned for paperspec (paperdesk PLAN M5.4,
paperform PLAN 4.2):

- paperspec correct and paperspec.revision apply proposed corrections to a
  package, folder or zip, and write it only once it validates.
- doc.math.form is checked: label and style commands always, the form's own
  spelling when papermath is installed or forms= is passed.
- paperspec.datasets and seven doc.dataset.* rules check a package's datasets.
- fixtures/invalid gains suspect-witness and math-form-label (33 in all).
- Canonical JSON reads numbers as doubles, so 1e20 round-trips.
- The registries and the JSON-LD context ship in the wheel.
- Public names for what paperdesk and paperform imported as private ones,
  with the old names kept as aliases.
- The text and lossy-projection functions are about three times faster
  through the gate, tested equal to the old code on every code point.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013XwDhn4LkrKQfw4ArDEr2E
2026-10-20 22:10:17 -04:00

34 lines
2.9 KiB
Markdown

# 15. Datasets
A dataset, `scienting-dataset/1`, is a Frictionless Data Package (`datapackage.json` with CSV or Parquet resources) with three additions.
| Addition | Meaning |
| --- | --- |
| Units | Every numeric field has a `unit`, a string Pint can parse, so a checker can test dimensions. |
| Uncertainty | A field may give `uncertainty`: `{field?, kind}`, the field that holds it and its kind, `sd`, `se`, `resolution` or `unknown`. |
| Origin | Each resource has an `origin`, or `origin_field` and `origins` when it varies by row. |
| Origin | Holds |
| --- | --- |
| `reported` | Values anchored to the table cells or sentences that print them, in `anchors.jsonl`. |
| `digitized` | Values read off a figure: the digitization's ID, and `anchors.jsonl` naming each value's series and point. |
| `computed` | A claim's implementation, by claim and pin. |
| `measured` | The team's own data, with its provenance. |
The descriptor also carries `format` and an `id`, `ds-` and a key.
Each line of `anchors.jsonl` names a `resource`, a `row` (by primary key) and a `field`, and either an `anchor` or a `digitization`, `series` and `point`.
A value or row is addressed as `ds-…/<row>/<field>`.
Values keep their digits as printed: CSV cells are never reformatted, and in JSON values are strings, so `1.526` and `1.5260` stay different.
A digitized value never passes as reported, and its uncertainty is at least the resolution of its digitization.
A dataset's content hash is SHA-256 of the canonical JSON of `{"format": "scienting-dataset/1", "files": [[path, sha256, size], …]}` over its files, sorted by path.
A dataset can be a source of kind `dataset`, so a claim can rest on a value read from a figure, with that fact visible in its evidence.
A validator checks each dataset of a package against the package.
It reports a descriptor that does not match `dataset.schema.json` (`doc.dataset.schema`), a folder not named by the dataset's ID (`doc.dataset.id`), a file that cannot be read (`doc.dataset.file`), and a CSV resource whose primary key is not unique (`doc.dataset.key`).
Each line of `anchors.jsonl` matches its schema and names a value of its resource, and its anchor resolves as current ([6.5](06-anchors.md#65-resolution)) (`doc.dataset.anchor`), to text that is the value as the CSV holds it once both are normalized ([1.4](01-conventions.md#14-text-normalization-for-hashing)) (`doc.dataset.value`).
Every numeric value of a `reported` resource is anchored (`doc.dataset.unanchored`); a resource's other fields, such as the labels that name its rows, need not be.
A correction that changes or removes an anchored cell makes these fail, so a producer that applies one rebuilds the dataset or leaves it out, as it does a derived file ([12.7](12-package.md#127-derived-files)).
`fixtures/packages/hale2026harmonic-damb/datasets/ds-hale-bond-params` reports Table 1, and the suite checks that every value resolves to the cell that prints it.