Initial commit

spec/00-introduction.md

@@ -0,0 +1,68 @@
+# 0. Introduction
+
+## 0.1 Scope
+
+These specifications define the data the Scientific Computing Studio's parts write and read: documents and their packages, anchors, receipts, rights records, layers, exercises, datasets, digitizations, views, projects and bundles.
+They define data, not programs.
+A conforming file can be read, checked and loaded without any studio code, and every rule here is stated so that a second implementation, in any language, computes the same bytes.
+
+The design document, *The Scientific Computing Studio: design of Lode, paperpack and Varve*, explains why each format exists.
+This text says exactly what each format is.
+Where the two differ, this text governs, and [DECISIONS.md](../DECISIONS.md) records why it differs.
+
+## 0.2 Conformance language
+
+The key words MUST, MUST NOT, REQUIRED, SHOULD, SHOULD NOT, RECOMMENDED, MAY and OPTIONAL are to be interpreted as described in RFC 2119 and RFC 8174 when, and only when, they appear in capitals.
+
+Three classes of implementation are distinguished.
+
+| Class | Does | Must |
+| --- | --- | --- |
+| Producer | Writes files in a format. | Write only files that validate, with values computed as specified. |
+| Consumer | Reads files in a format. | Accept every valid file, and refuse every file the suite lists as invalid, naming the rule. |
+| Validator | Decides whether a file is valid. | Check every rule in [22](22-conformance.md) that the file's contents allow it to check. |
+
+An implementation states, for each format, which versions it reads and writes.
+
+## 0.3 Format identifiers and versions
+
+Every format has an identifier of the form `scienting-<name>/<major>`, such as `scienting-doc/1`.
+A file or record that can travel on its own carries its identifier in a `format` property.
+Records inside a package file whose format the manifest fixes, such as the lines of `nodes.jsonl`, do not repeat it.
+
+A change that makes a valid file invalid, or that changes a computed value, is a breaking change and needs a new major version, with a migration.
+Adding an optional property, a node kind, a run kind or an enumeration value is a minor change.
+A consumer of version *n* MUST refuse a major version it does not know, and SHOULD warn on, but accept, enumeration values added in a later minor version.
+
+The specification as a whole is versioned with semantic versioning; this is version 0.1.0.
+[registry/formats.json](../registry/formats.json) lists every format with its media type, schema and status.
+
+## 0.4 Status
+
+Each format has one of three statuses.
+
+| Status | Meaning |
+| --- | --- |
+| `candidate` | Complete, with vectors and fixtures, and ready to implement. Changes before 1.0 need a recorded decision. |
+| `provisional` | Complete, but its milestone has not come, and it may change with less ceremony. |
+| `reserved` | Named and shaped, but not yet specified in full. |
+
+The design builds in levels, and level 1 implements a subset.
+Fields that level 1 does not use, such as rights beyond the license, receipts and spot-checks, are defined now so that formats never change when a team moves up a level.
+A consumer that does not use a field MUST still preserve it when it rewrites a file.
+
+## 0.5 How the repository is arranged
+
+| Path | Holds | Normative |
+| --- | --- | --- |
+| `spec/` | These chapters. | Yes |
+| `schemas/` | JSON Schemas, draft 2020-12. | Yes |
+| `sql/` | Table definitions for SQLite, Dolt and PostgreSQL, and the SQLite bundle. | Yes |
+| `context/` | The JSON-LD context for layers. | Yes |
+| `registry/` | Formats, rule IDs, stamp rules and the default stage requirements. | Yes |
+| `vectors/` | Test vectors, one case per line. | Yes |
+| `fixtures/` | Valid packages and records, invalid packages, and the keys that signed them. | Yes |
+| `tools/` | The oracle that computed the vectors, the builder and the checker. | No |
+
+The oracle exists so that no vector is typed by hand.
+It is not a library: implementations conform to the chapters and the vectors, not to the oracle, and `scienting-core` is the reference implementation.