First implementation (#1)
Reviewed-on: #1
This commit was merged in pull request #1.
This commit is contained in:
alexm committed
1 parent
34c0abb26a
commit
4fa326dc76
534 files changed
+28280
-37
No files matched your search
@@ -0,0 +1,86 @@
|
||||
# Registry
|
||||
|
||||
The registry holds lists that tools read at run time.
|
||||
Each list is a data file, so a reader, writer, converter or checker loads it instead of copying it into code, and a team can change a list without changing a program.
|
||||
This page is for newcomers, implementers who need the exact behavior, and experts who want the status of each format.
|
||||
|
||||
| File | Holds |
|
||||
| --- | --- |
|
||||
| `formats.json` | Every format: its identifier, media type (null for the six that have none), schema, status, chapter and a one-line description. |
|
||||
| `rules.json` | Every validation rule ID, with what it means and the chapter that defines it. |
|
||||
| `stamps/scienting-stamps-1.json` | The published rules that recognize the download stamps publishers add to PDFs (see Adding stamp rules), so page fingerprints ignore them. |
|
||||
| `stages/scienting-stages-default.json` | The rights each processing stage needs, as a default a team may replace. |
|
||||
|
||||
## Format status
|
||||
|
||||
Each entry in `formats.json` has a `status`.
|
||||
A `candidate` format is complete, with vectors and fixtures, and ready to implement.
|
||||
It changes only with a recorded reason and a new or changed vector.
|
||||
A `provisional` format is complete and expected to change as implementations gain experience, so it may change more freely.
|
||||
At version 0.1.0, 19 of the 32 formats are `candidate` and 13 are `provisional`.
|
||||
The third defined value, `reserved`, is unused.
|
||||
The values are defined in [Chapter 0, section 0.4](../spec/00-introduction.md#04-status), and the stability rules in [Chapter 22, section 22.5](../spec/22-conformance.md#225-stability).
|
||||
|
||||
## Registering the media types
|
||||
|
||||
The `application/vnd.scienting.*` media types are vendor-tree types, which IANA registers on request without a standards document (RFC 6838 §3.2).
|
||||
Register each type the formats serve over the web before the first release that other tools read: the zipped package, the anchor, the receipt and the layer first.
|
||||
Until then the types are unregistered and work as they are; a registration changes no byte of any file.
|
||||
The JSON-LD context is `application/ld+json` and a checkpoint is `text/plain`, both already registered.
|
||||
|
||||
## Rule IDs
|
||||
|
||||
A validator checks a package and reports each failure as `path:line: rule: message`, for example:
|
||||
|
||||
```text
|
||||
nodes.jsonl:6: doc.hash.node-hash: node_hash does not recompute
|
||||
```
|
||||
|
||||
A `node_hash` is a hash (a fixed-length value computed from content, so any change to the content changes it) stored with each node of a document.
|
||||
The line above is the failure that `fixtures/invalid/hash-mismatch/` is built to cause: its sixth node record carries a `node_hash` of all zeros.
|
||||
|
||||
A rule's meaning never changes, and a retired rule's ID is never reused, so tools and people can rely on the IDs.
|
||||
A tool can match on the `rule` field, whose meaning is looked up in `rules.json`, and treat the message as text for people.
|
||||
[Chapter 22, section 22.4](../spec/22-conformance.md#224-rules) lists the ID prefixes.
|
||||
Each directory in `fixtures/invalid/` has an `expect.json` that names the rule it breaks.
|
||||
The reference checker (`tools/check.py`) fails if that rule is not in `rules.json`.
|
||||
|
||||
## Adding stamp rules
|
||||
|
||||
Many publishers add a line to each PDF they serve, naming the institution and the time of download.
|
||||
This page calls that line a stamp.
|
||||
Two downloads of one paper then differ, so a hash of the file cannot match them.
|
||||
A page fingerprint is a hash of a page's size, its text and its images.
|
||||
It is computed after removing every line that a stamp rule matches ([Chapter 7, section 7.1](../spec/07-geometry.md#71-page-fingerprints)).
|
||||
The vectors in `vectors/fingerprints.jsonl` show this: `acs-download-1` and `acs-download-2` differ in the institution and the time on their stamp lines and have the same fingerprint.
|
||||
|
||||
Suppose two downloads of one paper differ only in a stamp line that no published rule recognizes.
|
||||
Record a rule in a file of your own, marked as learned:
|
||||
|
||||
```json
|
||||
{"format": "scienting-stamps/1", "id": "smithlab-learned", "learned": true,
|
||||
"rules": [{"id": "licensed-to", "pattern": "^This copy is licensed to .+", "where": "margin", "margin": 36}]}
|
||||
```
|
||||
|
||||
Normalized text is the line after Unicode Normalization Form C, with each run of whitespace replaced by one space and the ends trimmed ([Chapter 1, section 1.4](../spec/01-conventions.md#14-text-normalization-for-hashing)).
|
||||
|
||||
A rule has an `id`, a `pattern`, and a `where` that is `any` or `margin`.
|
||||
A `margin` rule matches only lines whose box lies wholly within `margin` points of one page edge (36 when the rule gives none).
|
||||
Patterns use only syntax that ECMAScript and Python read alike, and match a line's normalized text, with the ASCII letters A to Z compared without case.
|
||||
The allowed syntax is literals, `.`, `*`, `+`, `?`, `^`, `$`, classes and escaped punctuation.
|
||||
|
||||
A page record in `pages.jsonl` lists, in `stamps_removed`, the IDs of the rules that removed lines.
|
||||
A rule that is too broad hides a real difference between two pages.
|
||||
A stamp that no rule recognizes splits one edition into two ([Chapter 7, section 7.2](../spec/07-geometry.md#72-stamp-rules)).
|
||||
|
||||
## Replacing the stage defaults
|
||||
|
||||
A stage is one step of processing, such as conversion or indexing.
|
||||
Each stage needs certain rights, such as the right to mine text, and a stage runs only when the rights record of the paper allows them.
|
||||
The stage requirements are configuration, not code, because what a team may do depends on how it holds each paper.
|
||||
|
||||
A team copies `scienting-stages-default.json`, adjusts it with its counsel and points its project's settings at the copy.
|
||||
The file itself calls its contents "proposed defaults from the design, for counsel to set".
|
||||
In a project, the `settings.stages` field holds the `id` of the stage file in use ([Chapter 20](../spec/20-projects.md)), so a team's copy needs an `id` of its own.
|
||||
The project format is `provisional`.
|
||||
[Chapter 14](../spec/14-rights.md) explains how stages are checked.
|
||||
@@ -0,0 +1,262 @@
|
||||
{
|
||||
"registry": "formats",
|
||||
"version": "0.1.0",
|
||||
"formats": [
|
||||
{
|
||||
"format": "scienting-package/1",
|
||||
"media_type": "application/vnd.scienting.doc+zip",
|
||||
"schema": "manifest.schema.json",
|
||||
"status": "candidate",
|
||||
"chapter": "spec/12-package.md",
|
||||
"description": "A package: MANIFEST.json and its files, a folder or a zip."
|
||||
},
|
||||
{
|
||||
"format": "scienting-doc/1",
|
||||
"media_type": "application/vnd.scienting.doc+json",
|
||||
"schema": "document.schema.json",
|
||||
"status": "candidate",
|
||||
"chapter": "spec/02-model.md",
|
||||
"description": "A document: document.json, with nodes.jsonl and the package's other files."
|
||||
},
|
||||
{
|
||||
"format": "scienting-page/1",
|
||||
"media_type": null,
|
||||
"schema": "page.schema.json",
|
||||
"status": "candidate",
|
||||
"chapter": "spec/07-geometry.md",
|
||||
"description": "The page fingerprint function, and a page record."
|
||||
},
|
||||
{
|
||||
"format": "scienting-anchor/1",
|
||||
"media_type": "application/vnd.scienting.anchor+json",
|
||||
"schema": "anchor.schema.json",
|
||||
"status": "candidate",
|
||||
"chapter": "spec/06-anchors.md",
|
||||
"description": "A pinned address with enough context to survive change."
|
||||
},
|
||||
{
|
||||
"format": "scienting-correction/1",
|
||||
"media_type": "application/vnd.scienting.correction+json",
|
||||
"schema": "correction.schema.json",
|
||||
"status": "candidate",
|
||||
"chapter": "spec/11-corrections.md",
|
||||
"description": "An operation on an established document."
|
||||
},
|
||||
{
|
||||
"format": "scienting-receipt/1",
|
||||
"media_type": "application/vnd.scienting.receipt+json",
|
||||
"schema": "receipt.schema.json",
|
||||
"status": "candidate",
|
||||
"chapter": "spec/08-fidelity.md",
|
||||
"description": "A signature over a Merkle root of reviewed values."
|
||||
},
|
||||
{
|
||||
"format": "scienting-revocation/1",
|
||||
"media_type": "application/vnd.scienting.revocation+json",
|
||||
"schema": "revocation.schema.json",
|
||||
"status": "candidate",
|
||||
"chapter": "spec/08-fidelity.md",
|
||||
"description": "A signed withdrawal of receipts."
|
||||
},
|
||||
{
|
||||
"format": "scienting-roster/1",
|
||||
"media_type": "application/vnd.scienting.roster+json",
|
||||
"schema": "roster.schema.json",
|
||||
"status": "candidate",
|
||||
"chapter": "spec/08-fidelity.md",
|
||||
"description": "The keys that may sign, and for what."
|
||||
},
|
||||
{
|
||||
"format": "scienting-qa-report/1",
|
||||
"media_type": "application/vnd.scienting.qa-report+json",
|
||||
"schema": "qa-report.schema.json",
|
||||
"status": "candidate",
|
||||
"chapter": "spec/08-fidelity.md",
|
||||
"description": "Reviewer error bounds for a delivery, signed."
|
||||
},
|
||||
{
|
||||
"format": "scienting-spot-check/1",
|
||||
"media_type": null,
|
||||
"schema": "spot-check.schema.json",
|
||||
"status": "candidate",
|
||||
"chapter": "spec/08-fidelity.md",
|
||||
"description": "The input a spot-check key proves a VRF output for, to seed a sample; never a file of its own."
|
||||
},
|
||||
{
|
||||
"format": "scienting-proof/1",
|
||||
"media_type": "application/vnd.scienting.proof+json",
|
||||
"schema": "proof.schema.json",
|
||||
"status": "candidate",
|
||||
"chapter": "spec/08-fidelity.md",
|
||||
"description": "An RFC 9162 inclusion proof for one item of a receipt, or a consistency proof between two revisions."
|
||||
},
|
||||
{
|
||||
"format": "scienting-rights/1",
|
||||
"media_type": "application/vnd.scienting.rights+json",
|
||||
"schema": "rights.schema.json",
|
||||
"status": "candidate",
|
||||
"chapter": "spec/14-rights.md",
|
||||
"description": "A witness's license, basis and uses."
|
||||
},
|
||||
{
|
||||
"format": "scienting-policy/1",
|
||||
"media_type": "application/vnd.scienting.policy+json",
|
||||
"schema": "policy.schema.json",
|
||||
"status": "candidate",
|
||||
"chapter": "spec/14-rights.md",
|
||||
"description": "A signed team decision that resolves unknown uses."
|
||||
},
|
||||
{
|
||||
"format": "scienting-stages/1",
|
||||
"media_type": "application/vnd.scienting.stages+json",
|
||||
"schema": "stages.schema.json",
|
||||
"status": "candidate",
|
||||
"chapter": "spec/14-rights.md",
|
||||
"description": "The uses each stage needs."
|
||||
},
|
||||
{
|
||||
"format": "scienting-backend/1",
|
||||
"media_type": "application/vnd.scienting.backend+json",
|
||||
"schema": "backend.schema.json",
|
||||
"status": "candidate",
|
||||
"chapter": "spec/14-rights.md",
|
||||
"description": "A tool's code and weight licenses and their conditions."
|
||||
},
|
||||
{
|
||||
"format": "scienting-profile/1",
|
||||
"media_type": "application/vnd.scienting.profile+json",
|
||||
"schema": "profile.schema.json",
|
||||
"status": "candidate",
|
||||
"chapter": "spec/14-rights.md",
|
||||
"description": "What backend conditions ask about a team."
|
||||
},
|
||||
{
|
||||
"format": "scienting-stamps/1",
|
||||
"media_type": "application/vnd.scienting.stamps+json",
|
||||
"schema": "stamps.schema.json",
|
||||
"status": "candidate",
|
||||
"chapter": "spec/07-geometry.md",
|
||||
"description": "Download-stamp line rules."
|
||||
},
|
||||
{
|
||||
"format": "scienting-suspect/1",
|
||||
"media_type": null,
|
||||
"schema": "suspect.schema.json",
|
||||
"status": "candidate",
|
||||
"chapter": "spec/05-hashes.md",
|
||||
"description": "A doubtful reading for review."
|
||||
},
|
||||
{
|
||||
"format": "scienting-layer/1",
|
||||
"media_type": "application/ld+json; profile=\"http://www.w3.org/ns/anno.jsonld\"",
|
||||
"schema": "layer.schema.json",
|
||||
"status": "candidate",
|
||||
"chapter": "spec/18-layers.md",
|
||||
"description": "A layer of W3C Web Annotations."
|
||||
},
|
||||
{
|
||||
"format": "scienting-exercise/1",
|
||||
"media_type": null,
|
||||
"schema": "exercise.schema.json",
|
||||
"status": "provisional",
|
||||
"chapter": "spec/19-exercises.md",
|
||||
"description": "The body of an exercise annotation."
|
||||
},
|
||||
{
|
||||
"format": "scienting-submission/1",
|
||||
"media_type": "application/vnd.scienting.submission+json",
|
||||
"schema": "submission.schema.json",
|
||||
"status": "provisional",
|
||||
"chapter": "spec/19-exercises.md",
|
||||
"description": "A student's answer under a pseudonym."
|
||||
},
|
||||
{
|
||||
"format": "scienting-grade/1",
|
||||
"media_type": "application/vnd.scienting.grade+json",
|
||||
"schema": "grade.schema.json",
|
||||
"status": "provisional",
|
||||
"chapter": "spec/19-exercises.md",
|
||||
"description": "The grader's result."
|
||||
},
|
||||
{
|
||||
"format": "scienting-project/1",
|
||||
"media_type": "application/vnd.scienting.project+json",
|
||||
"schema": "project.schema.json",
|
||||
"status": "provisional",
|
||||
"chapter": "spec/20-projects.md",
|
||||
"description": "A project."
|
||||
},
|
||||
{
|
||||
"format": "scienting-bundle/1",
|
||||
"media_type": "application/vnd.sqlite3",
|
||||
"schema": "sql/bundle-sqlite.sql",
|
||||
"status": "provisional",
|
||||
"chapter": "spec/13-database.md",
|
||||
"description": "The SQLite bundle readers and servers query."
|
||||
},
|
||||
{
|
||||
"format": "scienting-dataset/1",
|
||||
"media_type": "application/vnd.scienting.dataset+json",
|
||||
"schema": "dataset.schema.json",
|
||||
"status": "provisional",
|
||||
"chapter": "spec/15-datasets.md",
|
||||
"description": "A Frictionless Data Package with units, uncertainty and origin."
|
||||
},
|
||||
{
|
||||
"format": "scienting-digitization/1",
|
||||
"media_type": "application/vnd.scienting.digitization+json",
|
||||
"schema": "digitization.schema.json",
|
||||
"status": "provisional",
|
||||
"chapter": "spec/16-digitization.md",
|
||||
"description": "How values were read off a figure panel."
|
||||
},
|
||||
{
|
||||
"format": "scienting-view/1",
|
||||
"media_type": "application/vnd.vegalite+json",
|
||||
"schema": "view.schema.json",
|
||||
"status": "provisional",
|
||||
"chapter": "spec/17-views.md",
|
||||
"description": "A Vega-Lite view that pins its data."
|
||||
},
|
||||
{
|
||||
"format": "scienting-snapshot/1",
|
||||
"media_type": "application/vnd.scienting.snapshot+json",
|
||||
"schema": "snapshot.schema.json",
|
||||
"status": "provisional",
|
||||
"chapter": "spec/21-drafts.md",
|
||||
"description": "What a draft cites, pinned."
|
||||
},
|
||||
{
|
||||
"format": "scienting-draft-ids/1",
|
||||
"media_type": null,
|
||||
"schema": "draft-ids.schema.json",
|
||||
"status": "provisional",
|
||||
"chapter": "spec/21-drafts.md",
|
||||
"description": "A Typst project's sentence-ID sidecar."
|
||||
},
|
||||
{
|
||||
"format": "scienting-checklist/1",
|
||||
"media_type": "application/vnd.scienting.checklist+json",
|
||||
"schema": "checklist.schema.json",
|
||||
"status": "provisional",
|
||||
"chapter": "spec/08-fidelity.md",
|
||||
"description": "A check of an export against a named standard, covered by a receipt."
|
||||
},
|
||||
{
|
||||
"format": "scienting-log-entry/1",
|
||||
"media_type": null,
|
||||
"schema": "log-entry.schema.json",
|
||||
"status": "provisional",
|
||||
"chapter": "spec/23-log.md",
|
||||
"description": "One item of a team's transparency log: a receipt, revocation or roster by kind, ID and hash."
|
||||
},
|
||||
{
|
||||
"format": "scienting-checkpoint/1",
|
||||
"media_type": "text/plain; charset=utf-8",
|
||||
"schema": "checkpoint.schema.json",
|
||||
"status": "provisional",
|
||||
"chapter": "spec/23-log.md",
|
||||
"description": "A signed checkpoint of a team's log: a signed note with the origin, size and root."
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,497 @@
|
||||
{
|
||||
"registry": "rules",
|
||||
"version": "0.1.0",
|
||||
"note": "Rule IDs name each refusal, so tools report file, node and rule in the form path:line: rule: message.",
|
||||
"rules": [
|
||||
{
|
||||
"id": "correction.digits",
|
||||
"description": "A digit changes without crop evidence, or a leaf's digits change undeclared.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "correction.evidence",
|
||||
"description": "An insert has no crop evidence.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "correction.find",
|
||||
"description": "`find` is empty or does not occur exactly once.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "correction.insert",
|
||||
"description": "Text is inserted outside the runs of a node, or at a bad offset.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "correction.merge",
|
||||
"description": "The nodes differ in kind, are not mergeable, or are not adjacent siblings.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "correction.move",
|
||||
"description": "The destination is unknown or inside the moved node.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "correction.op",
|
||||
"description": "The operation is unknown.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "correction.reclassify",
|
||||
"description": "The new kind has another content shape.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "correction.relabel",
|
||||
"description": "The kind takes no label.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "correction.replace",
|
||||
"description": "Runs replace text outside a text run, or include an equation run.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "correction.schema",
|
||||
"description": "A proposed correction does not match the correction schema.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "correction.set",
|
||||
"description": "The attribute cannot be set on the kind.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "correction.split",
|
||||
"description": "The node is not a sentence, the offset is outside it or inside a run other than text, or a part is empty.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "correction.target",
|
||||
"description": "A target, parent or sibling is not a live node, or the root is deleted, moved, merged or reclassified.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.correction.after",
|
||||
"description": "The last correction of a leaf does not end at the leaf's current node_hash.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.correction.id",
|
||||
"description": "A correction's id does not derive from its record.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.correction.index",
|
||||
"description": "Correction indexes do not count from 1 without gaps.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.correction.rev",
|
||||
"description": "document.json's rev is not the number of corrections.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.fidelity.report",
|
||||
"description": "document.json's fidelity distribution does not recompute.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.hash.align-key",
|
||||
"description": "An align_key does not recompute.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.hash.algorithm",
|
||||
"description": "document.json names no hash algorithm, or one the consumer does not implement.",
|
||||
"chapter": "spec/01-conventions.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.roster.did",
|
||||
"description": "A roster key's did:key is not the one its public key determines.",
|
||||
"chapter": "spec/08-fidelity.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.hash.key",
|
||||
"description": "The key supplied does not match the package's key_id.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.hash.match-hash",
|
||||
"description": "A match_hash does not recompute.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.hash.meta",
|
||||
"description": "meta_hash does not recompute.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.hash.node-hash",
|
||||
"description": "A node_hash does not recompute.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.hash.own-hash",
|
||||
"description": "An own_hash does not recompute.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.hash.root",
|
||||
"description": "document.json's root is not the root node's node_hash.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.hash.text-hash",
|
||||
"description": "A text_hash does not recompute.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.id.alias",
|
||||
"description": "An alias does not name a live node.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.id.form",
|
||||
"description": "An ID is not seven letter-first Crockford base32 characters.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.id.retired-live",
|
||||
"description": "An ID is both live and in the ID table.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.id.retired-missing",
|
||||
"description": "A correction retired an ID that ids.jsonl does not record.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.id.successor",
|
||||
"description": "A successor names an ID that is neither live nor in the ID table.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.id.unique",
|
||||
"description": "An ID occurs twice.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.location.node",
|
||||
"description": "A location names an unknown node.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.location.page",
|
||||
"description": "A location's page_fp is not a page of any witness.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.node.attrs",
|
||||
"description": "A section has no level.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.node.content",
|
||||
"description": "A node's content does not have the shape its kind requires.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.node.kind",
|
||||
"description": "A node kind is not in the list.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.node.label",
|
||||
"description": "A node of a kind that takes no label has one.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.node.schema",
|
||||
"description": "A node a correction produced does not validate against the node schema (a property the kind does not take, or a bad shape).",
|
||||
"chapter": "spec/11-corrections.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.counts",
|
||||
"description": "document.json's counts are not the number of nodes and of leaves.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.history",
|
||||
"description": "document.json's history is not the root of the tree over the package's corrections.",
|
||||
"chapter": "spec/11-corrections.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.key.work",
|
||||
"description": "The document key does not begin with the work key, or its suffix is not a version suffix.",
|
||||
"chapter": "spec/02-model.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.key.suffix",
|
||||
"description": "The work key's hash part is not the hash of the work's identifier.",
|
||||
"chapter": "spec/02-model.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.order.ids",
|
||||
"description": "ids.jsonl is not aliases by ID, then retirements by revision and ID.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.order.locations",
|
||||
"description": "locations.jsonl is not sorted by node order, page and fingerprint.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.order.pages",
|
||||
"description": "pages.jsonl is not sorted by witness and page.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.order.receipts",
|
||||
"description": "receipts.jsonl is not sorted by time and ID.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.order.references",
|
||||
"description": "references.jsonl does not follow document order.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.order.suspects",
|
||||
"description": "suspects.jsonl is not sorted by ID.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.page.witness",
|
||||
"description": "A page names an unknown witness.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.policy.keyed",
|
||||
"description": "A checks-only package carries plain hashes.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.policy.withheld",
|
||||
"description": "Text is present where the policy withholds it, or withheld where the policy is full, or a checks-only package carries corrections or suspects.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.qa.key",
|
||||
"description": "A spot-check seed key is not a registered Ed25519 spot-check key, or was registered after the receipt.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.qa.of",
|
||||
"description": "A spot-check names a receipt the package does not hold.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.qa.sample",
|
||||
"description": "A spot-check sample does not follow from its seed.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.qa.seed",
|
||||
"description": "A spot-check seed is not the checker's signature over the receipt.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.receipt.doc",
|
||||
"description": "A receipt names another document.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.receipt.id",
|
||||
"description": "A receipt's or revocation's id does not derive from its record.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.receipt.nodes",
|
||||
"description": "A receipt's nodes are not sorted by ID, or one repeats.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.receipt.root",
|
||||
"description": "A receipt's Merkle root does not recompute.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.receipt.signature",
|
||||
"description": "A signature does not verify against the roster.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.reference.node",
|
||||
"description": "A reference record names a node that is not a reference.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.run.canonical",
|
||||
"description": "Runs are not canonical: an empty text run, or two adjacent text runs with the same styles.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.run.cite",
|
||||
"description": "A cite run names a node that is not a reference.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.run.equation",
|
||||
"description": "Equation runs do not name the sentence's equation children, once each, in order.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.run.fnmark",
|
||||
"description": "An fnmark run names a node that is not a footnote.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.run.kind",
|
||||
"description": "A run kind is not in the list.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.run.style",
|
||||
"description": "A style is unknown, or a run is both a subscript and a superscript.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.run.xref",
|
||||
"description": "A same-document xref names no node.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.table.header",
|
||||
"description": "A column or row header path names a node that is not a cell.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.tree.child",
|
||||
"description": "A node holds a child kind it may not hold, a heading is not first, or a once-only child repeats.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.tree.empty",
|
||||
"description": "A kind that must have children has none.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.tree.ord",
|
||||
"description": "A parent's children do not have ord 0 to n-1.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.tree.order",
|
||||
"description": "nodes.jsonl is not in document order.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.tree.root",
|
||||
"description": "The tree does not have exactly one root, of kind document.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "doc.witness.id",
|
||||
"description": "A witness ID does not derive from its SHA-256.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "grade.key-stale",
|
||||
"description": "An anchor in the exercise's key is stale, so nothing is graded with it.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "grade.kind",
|
||||
"description": "The exercise kind is unknown.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "package.file.canonical",
|
||||
"description": "A JSON or JSON Lines record is not in canonical JSON.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "package.file.empty",
|
||||
"description": "An optional JSON Lines file is present with no records.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "package.file.newline",
|
||||
"description": "A line of a JSON Lines file does not end with a newline.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "package.file.required",
|
||||
"description": "document.json, nodes.jsonl, witnesses.json or provenance.json is missing.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "package.manifest.absent",
|
||||
"description": "An entry listed without its bytes is not an asset, or its bytes are present.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "package.manifest.hash",
|
||||
"description": "A file's SHA-256 or size differs from its manifest entry.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "package.manifest.key",
|
||||
"description": "The manifest's key differs from document.json's.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "package.manifest.missing",
|
||||
"description": "The package has no MANIFEST.json.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "package.manifest.missing-file",
|
||||
"description": "A file the manifest lists is not in the package.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "package.manifest.order",
|
||||
"description": "Manifest entries are not sorted by path, or a path repeats.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "package.manifest.schema",
|
||||
"description": "MANIFEST.json does not validate against manifest.schema.json.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "package.manifest.unlisted",
|
||||
"description": "The package holds a file the manifest does not list.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "package.path.unsafe",
|
||||
"description": "A path is absolute, has an empty, `.` or `..` segment, a backslash or a NUL, is not normalized, or is or passes through a symbolic link.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "package.size",
|
||||
"description": "The package's files exceed the consumer's size limit.",
|
||||
"chapter": "spec/12-package.md"
|
||||
},
|
||||
{
|
||||
"id": "package.schema",
|
||||
"description": "A file or line does not validate against its schema.",
|
||||
"chapter": "spec/22-conformance.md"
|
||||
},
|
||||
{
|
||||
"id": "package.zip.entry",
|
||||
"description": "A zipped package is not a zip archive, or an entry is encrypted, a symbolic link, neither stored nor deflated, not marked as UTF-8, duplicated, unsafe as a path, or not under one top-level folder named by the key.",
|
||||
"chapter": "spec/12-package.md"
|
||||
},
|
||||
{
|
||||
"id": "package.zip.size",
|
||||
"description": "A zipped package expands past the consumer's limit, or an entry holds more than its header declares.",
|
||||
"chapter": "spec/12-package.md"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,85 @@
|
||||
{
|
||||
"format": "scienting-stages/1",
|
||||
"id": "scienting-stages-default",
|
||||
"note": "Proposed defaults from the design, for counsel to set. Configuration, not code: a team may replace this file.",
|
||||
"stages": {
|
||||
"convert-model": {
|
||||
"uses": [
|
||||
"tdm"
|
||||
],
|
||||
"ai": {
|
||||
"purpose": "internal"
|
||||
},
|
||||
"note": "Conversion by any model, local or outside."
|
||||
},
|
||||
"convert-deterministic": {
|
||||
"uses": [
|
||||
"tdm"
|
||||
],
|
||||
"note": "The text-layer witness, segmentation, hashing and the gate."
|
||||
},
|
||||
"index": {
|
||||
"uses": [
|
||||
"tdm"
|
||||
],
|
||||
"note": "The full-text index."
|
||||
},
|
||||
"review-model": {
|
||||
"uses": [],
|
||||
"ai": {
|
||||
"purpose": "internal"
|
||||
},
|
||||
"note": "Model review of suspects, in a job zip or the direct API mode."
|
||||
},
|
||||
"propose-model": {
|
||||
"uses": [],
|
||||
"ai": {
|
||||
"purpose": "internal"
|
||||
},
|
||||
"note": "Glosses, digitizations or claims a model proposes."
|
||||
},
|
||||
"mcp-text": {
|
||||
"uses": [],
|
||||
"ai": {
|
||||
"purpose": "internal"
|
||||
},
|
||||
"note": "A text-serving interface returning text to the team's model."
|
||||
},
|
||||
"embed": {
|
||||
"uses": [],
|
||||
"ai": {
|
||||
"purpose": "internal",
|
||||
"processors": [
|
||||
"local",
|
||||
"tenancy"
|
||||
]
|
||||
},
|
||||
"note": "Embeddings for search."
|
||||
},
|
||||
"ai-data-external": {
|
||||
"uses": [],
|
||||
"ai": {
|
||||
"purpose": "external"
|
||||
},
|
||||
"note": "AI data built for others; in practice CC BY papers only."
|
||||
},
|
||||
"read": {
|
||||
"uses": [
|
||||
"read"
|
||||
],
|
||||
"note": "Reading in the reader."
|
||||
},
|
||||
"publish-quotes": {
|
||||
"uses": [
|
||||
"quote-public"
|
||||
],
|
||||
"note": "Publishing anything with quotations."
|
||||
},
|
||||
"accessible-copy": {
|
||||
"uses": [
|
||||
"accessible-copy"
|
||||
],
|
||||
"note": "An accessible copy for a user with a disability."
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
{
|
||||
"format": "scienting-stamps/1",
|
||||
"id": "scienting-stamps-1",
|
||||
"learned": false,
|
||||
"note": "Published stamp-line rules, version 1. Patterns are matched against the normalized line, case-insensitively over ASCII. Teams add learned rules in files of their own.",
|
||||
"rules": [
|
||||
{
|
||||
"id": "downloaded-via",
|
||||
"pattern": "^Downloaded via .+ on .+",
|
||||
"where": "any",
|
||||
"note": "The institution-and-time stamp some publishers print in the margin."
|
||||
},
|
||||
{
|
||||
"id": "downloaded-from-on",
|
||||
"pattern": "^Downloaded from .+ on .+",
|
||||
"where": "any",
|
||||
"note": "The URL, institution and date stamp other publishers print."
|
||||
},
|
||||
{
|
||||
"id": "sharing-guidelines",
|
||||
"pattern": "^See https?://pubs\\.acs\\.org/sharingguidelines",
|
||||
"where": "any",
|
||||
"note": "A sharing notice printed beside a download stamp."
|
||||
},
|
||||
{
|
||||
"id": "terms-and-conditions",
|
||||
"pattern": "^See the Terms and Conditions \\(https?://onlinelibrary\\.wiley\\.com/",
|
||||
"where": "any",
|
||||
"note": "The rest of a stamp line, when a text layer splits it."
|
||||
},
|
||||
{
|
||||
"id": "margin-downloaded",
|
||||
"pattern": "^Downloaded",
|
||||
"where": "margin",
|
||||
"margin": 36,
|
||||
"note": "Any line in the outer 36 points that begins with Downloaded."
|
||||
}
|
||||
]
|
||||
}
|
||||
Reference in new issue
Block a user