Files
paperspec/spec/15-datasets.md
T
alexmandClaude 4fb8b4f5a7 Add paperspec correct, check math forms and datasets, and give the tools public names
The work the other paper tools planned for paperspec (paperdesk PLAN M5.4,
paperform PLAN 4.2):

- paperspec correct and paperspec.revision apply proposed corrections to a
  package, folder or zip, and write it only once it validates.
- doc.math.form is checked: label and style commands always, the form's own
  spelling when papermath is installed or forms= is passed.
- paperspec.datasets and seven doc.dataset.* rules check a package's datasets.
- fixtures/invalid gains suspect-witness and math-form-label (33 in all).
- Canonical JSON reads numbers as doubles, so 1e20 round-trips.
- The registries and the JSON-LD context ship in the wheel.
- Public names for what paperdesk and paperform imported as private ones,
  with the old names kept as aliases.
- The text and lossy-projection functions are about three times faster
  through the gate, tested equal to the old code on every code point.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013XwDhn4LkrKQfw4ArDEr2E
2026-10-20 22:10:17 -04:00

2.9 KiB

15. Datasets

A dataset, scienting-dataset/1, is a Frictionless Data Package (datapackage.json with CSV or Parquet resources) with three additions.

Addition Meaning
Units Every numeric field has a unit, a string Pint can parse, so a checker can test dimensions.
Uncertainty A field may give uncertainty: {field?, kind}, the field that holds it and its kind, sd, se, resolution or unknown.
Origin Each resource has an origin, or origin_field and origins when it varies by row.
Origin Holds
reported Values anchored to the table cells or sentences that print them, in anchors.jsonl.
digitized Values read off a figure: the digitization's ID, and anchors.jsonl naming each value's series and point.
computed A claim's implementation, by claim and pin.
measured The team's own data, with its provenance.

The descriptor also carries format and an id, ds- and a key. Each line of anchors.jsonl names a resource, a row (by primary key) and a field, and either an anchor or a digitization, series and point. A value or row is addressed as ds-…/<row>/<field>.

Values keep their digits as printed: CSV cells are never reformatted, and in JSON values are strings, so 1.526 and 1.5260 stay different. A digitized value never passes as reported, and its uncertainty is at least the resolution of its digitization.

A dataset's content hash is SHA-256 of the canonical JSON of {"format": "scienting-dataset/1", "files": [[path, sha256, size], …]} over its files, sorted by path. A dataset can be a source of kind dataset, so a claim can rest on a value read from a figure, with that fact visible in its evidence. A validator checks each dataset of a package against the package. It reports a descriptor that does not match dataset.schema.json (doc.dataset.schema), a folder not named by the dataset's ID (doc.dataset.id), a file that cannot be read (doc.dataset.file), and a CSV resource whose primary key is not unique (doc.dataset.key). Each line of anchors.jsonl matches its schema and names a value of its resource, and its anchor resolves as current (6.5) (doc.dataset.anchor), to text that is the value as the CSV holds it once both are normalized (1.4) (doc.dataset.value). Every numeric value of a reported resource is anchored (doc.dataset.unanchored); a resource's other fields, such as the labels that name its rows, need not be. A correction that changes or removes an anchored cell makes these fail, so a producer that applies one rebuilds the dataset or leaves it out, as it does a derived file (12.7).

fixtures/packages/hale2026harmonic-damb/datasets/ds-hale-bond-params reports Table 1, and the suite checks that every value resolves to the cell that prints it.