The work the other paper tools planned for paperspec (paperdesk PLAN M5.4, paperform PLAN 4.2): - paperspec correct and paperspec.revision apply proposed corrections to a package, folder or zip, and write it only once it validates. - doc.math.form is checked: label and style commands always, the form's own spelling when papermath is installed or forms= is passed. - paperspec.datasets and seven doc.dataset.* rules check a package's datasets. - fixtures/invalid gains suspect-witness and math-form-label (33 in all). - Canonical JSON reads numbers as doubles, so 1e20 round-trips. - The registries and the JSON-LD context ship in the wheel. - Public names for what paperdesk and paperform imported as private ones, with the old names kept as aliases. - The text and lossy-projection functions are about three times faster through the gate, tested equal to the old code on every code point. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013XwDhn4LkrKQfw4ArDEr2E
34 lines
2.9 KiB
Markdown
34 lines
2.9 KiB
Markdown
# 15. Datasets
|
|
|
|
A dataset, `scienting-dataset/1`, is a Frictionless Data Package (`datapackage.json` with CSV or Parquet resources) with three additions.
|
|
|
|
| Addition | Meaning |
|
|
| --- | --- |
|
|
| Units | Every numeric field has a `unit`, a string Pint can parse, so a checker can test dimensions. |
|
|
| Uncertainty | A field may give `uncertainty`: `{field?, kind}`, the field that holds it and its kind, `sd`, `se`, `resolution` or `unknown`. |
|
|
| Origin | Each resource has an `origin`, or `origin_field` and `origins` when it varies by row. |
|
|
|
|
| Origin | Holds |
|
|
| --- | --- |
|
|
| `reported` | Values anchored to the table cells or sentences that print them, in `anchors.jsonl`. |
|
|
| `digitized` | Values read off a figure: the digitization's ID, and `anchors.jsonl` naming each value's series and point. |
|
|
| `computed` | A claim's implementation, by claim and pin. |
|
|
| `measured` | The team's own data, with its provenance. |
|
|
|
|
The descriptor also carries `format` and an `id`, `ds-` and a key.
|
|
Each line of `anchors.jsonl` names a `resource`, a `row` (by primary key) and a `field`, and either an `anchor` or a `digitization`, `series` and `point`.
|
|
A value or row is addressed as `ds-…/<row>/<field>`.
|
|
|
|
Values keep their digits as printed: CSV cells are never reformatted, and in JSON values are strings, so `1.526` and `1.5260` stay different.
|
|
A digitized value never passes as reported, and its uncertainty is at least the resolution of its digitization.
|
|
|
|
A dataset's content hash is SHA-256 of the canonical JSON of `{"format": "scienting-dataset/1", "files": [[path, sha256, size], …]}` over its files, sorted by path.
|
|
A dataset can be a source of kind `dataset`, so a claim can rest on a value read from a figure, with that fact visible in its evidence.
|
|
A validator checks each dataset of a package against the package.
|
|
It reports a descriptor that does not match `dataset.schema.json` (`doc.dataset.schema`), a folder not named by the dataset's ID (`doc.dataset.id`), a file that cannot be read (`doc.dataset.file`), and a CSV resource whose primary key is not unique (`doc.dataset.key`).
|
|
Each line of `anchors.jsonl` matches its schema and names a value of its resource, and its anchor resolves as current ([6.5](06-anchors.md#65-resolution)) (`doc.dataset.anchor`), to text that is the value as the CSV holds it once both are normalized ([1.4](01-conventions.md#14-text-normalization-for-hashing)) (`doc.dataset.value`).
|
|
Every numeric value of a `reported` resource is anchored (`doc.dataset.unanchored`); a resource's other fields, such as the labels that name its rows, need not be.
|
|
A correction that changes or removes an anchored cell makes these fail, so a producer that applies one rebuilds the dataset or leaves it out, as it does a derived file ([12.7](12-package.md#127-derived-files)).
|
|
|
|
`fixtures/packages/hale2026harmonic-damb/datasets/ds-hale-bond-params` reports Table 1, and the suite checks that every value resolves to the cell that prints it.
|