Chemical structures (spec/18 section 18.6) and biological sequences (18.7) tied to
the words of a paper. Minor change: two new body kinds, optional properties and
seven rules, with no valid file made invalid.
- annotation schema: structure (SMILES or SMARTS, role, identity, written, label,
accessions, force-field terms) and sequence (alphabet, role, identity, start,
organism, regions, variants); body_kind in sql/store.sql; nine terms in the
JSON-LD context; rules layer.structure.{value,identity,term} and
layer.sequence.{value,identity,region,variant}.
- paperspec.structures reads the atoms and bonds of a SMILES or SMARTS with the
standard library, so a term's atoms are checked without a toolkit; checked against
RDKit on a corpus and random texts. paperspec.sequences checks alphabets,
identities, regions and variants.
- fixtures/records/chemistry (on the main example) and biology (on the draft, which
gains a last sentence), vectors/structures.jsonl and vectors/sequences.jsonl;
check.py runs both layers and the vectors; counts, guides and CHANGELOG updated.
- the prose check no longer takes organism for a UK spelling.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B8cL4zN4XpaQF5cBtASzfq
201 lines
21 KiB
Markdown
201 lines
21 KiB
Markdown
# 18. Layers and annotations
|
||
|
||
Everything people and models add to a document is an annotation, and annotations are grouped in layers.
|
||
`scienting-layer/1` is the W3C Web Annotation Data Model with the anchor as a selector, so any Web Annotation tool can read it.
|
||
The JSON-LD context is [context/scienting-1.jsonld](../context/scienting-1.jsonld), published at `https://paperspec.scient.ing/context/scienting-1.jsonld`.
|
||
|
||
## 18.1 One rule decides what is a claim and what is a layer
|
||
|
||
If a statement is true independently of this paper, it belongs in a claim store as a claim, where it is checked and leveled.
|
||
If it is about reading this paper, it belongs in a layer, where it is attributed and signed but not checked.
|
||
|
||
## 18.2 The layer
|
||
|
||
A layer is a folder with `layer.json` and `annotations.jsonl`.
|
||
`layer.json` is an `AnnotationCollection` with the two contexts, `format`, an `id` of the form `tag:scient.ing,2026:layer:<store>/<record id>`, a `label`, a `purpose`, an `audience` (`novice`, `student`, `expert` or `mixed`), a `visibility` (`private`, `team` or `public`), a `license` (CC BY 4.0 by default when public), its `creator`, `created`, its `project`, the `docs` it annotates and its `total`.
|
||
A reader turns layers on and off, and the audience dial picks the level of glosses.
|
||
|
||
## 18.3 The annotation
|
||
|
||
Each line of `annotations.jsonl` is a W3C `Annotation`:
|
||
|
||
| Property | Holds |
|
||
| --- | --- |
|
||
| `@context` | `["http://www.w3.org/ns/anno.jsonld", "https://paperspec.scient.ing/context/scienting-1.jsonld"]`. |
|
||
| `id` | `tag:scient.ing,2026:annotation:<store>/<record id>` ([4.6](04-identity.md#46-record-ids)). |
|
||
| `type`, `layer` | `Annotation`, and the layer's ID. |
|
||
| `motivation` | A W3C motivation: `commenting`, `describing`, `linking`, `questioning`, `assessing`, `replying` and the others. |
|
||
| `creator`, `created`, `modified` | An agent `{id, type, name}`, with `type` `Person` or `Software`, and times. |
|
||
| `body` | One body ([18.4](#184-bodies)). |
|
||
| `target` | A `SpecificResource` whose `source` is `tag:scient.ing,2026:doc:<store>/<key>` and whose `selector` lists an `AnchorSelector` holding a `scienting-anchor/1`, optionally followed by a W3C `TextQuoteSelector` for tools that do not know ours; or, for a reply, a `SpecificResource` whose source is another annotation. |
|
||
| `status` | `proposed`, `confirmed`, `rejected` or `withdrawn`. A model's annotation is `proposed` until a person confirms it. |
|
||
| `receipt` | For a confirmed annotation, the receipt that covers it. |
|
||
|
||
The `TextQuoteSelector`, when present, repeats the anchor quote's `exact`, `prefix` and `suffix`, with an empty string for a `prefix` or `suffix` the quote lacks.
|
||
It is present only when the anchor's quote is plain text; an anchor whose quote is hashed or keyed ([6.3](06-anchors.md#63-keyed-anchors)) has none, so the fallback never shows text the anchor withholds.
|
||
|
||
An annotation's content hash is SHA-256 of the canonical JSON of the annotation without `status`, `receipt` and `modified`.
|
||
A confirmed annotation is covered by its confirmer's receipt, which lists `["annotation", <record id>, <content hash>]`.
|
||
Replies are annotations that target annotations, so discussions are threads anchored to sentences.
|
||
|
||
## 18.4 Bodies
|
||
|
||
| `kind` | W3C type | Adds | Says |
|
||
| --- | --- | --- | --- |
|
||
| `gloss` | `TextualBody` | `audience` | An explanation at `novice`, `student` or `expert` level. |
|
||
| `claim-link` | `SpecificResource` | `source`, `pin`, `role` | That the span `states`, `uses` or `disputes` a claim. |
|
||
| `symbol` | `SpecificResource` | `source`, `pin`, `key`, `symbol` | That this occurrence of a symbol, the one whose key is `key`, is the symbol a symbol claim defines. |
|
||
| `comment` | `TextualBody` | `category` | A remark, opinion or interpretation. |
|
||
| `caveat` | `TextualBody` | `category` | A warning a reader should see. |
|
||
| `question` | `TextualBody` | `for`, `category` | A question for a `reader`, a `group` or the `authors`. |
|
||
| `exercise` | `TextualBody` | `exercise` | A task with a key and a rule ([19](19-exercises.md)). |
|
||
| `citation-context` | `TextualBody` | `reference`, `verdict`, `reason` | Why a sentence cites a reference, and whether it supports it. |
|
||
| `prerequisite` | `SpecificResource` | `source` | What to read or know first. |
|
||
| `crosslink` | `SpecificResource` | `source`, `selector` | That the span corresponds to a span in another document. |
|
||
| `trace` | `SpecificResource` | `source`, `selector`, `reference`, `note`, `verdict`, `proposer` | That the target's sentence cites `reference` on the strength of a passage of the cited document ([18.5](#185-traced-citations)). |
|
||
| `structure` | `TextualBody` | `format`, `role`, `identity`, `written`, `label`, `accessions`, `terms` | That the span names, draws, gives parameters for or uses a chemical structure ([18.6](#186-chemical-structures)). |
|
||
| `sequence` | `TextualBody` | `alphabet`, `role`, `identity`, `start`, `label`, `organism`, `accessions`, `regions`, `variants` | That the span names, shows, uses or reports a variant of a protein, DNA or RNA sequence ([18.7](#187-sequences)). |
|
||
| `translation` | `TextualBody` | `language` | The span in another language. |
|
||
| `description` | `TextualBody` | `role` | A description of a figure, panel or table for readers who cannot see it: `alt`, a short text equivalent, or `long`, a full description. |
|
||
| `generated` | `TextualBody` | `generator` | Text a model wrote about the span; never mistaken for source. `generator` is `{model, run}` as a correction's model author is ([11.2](11-corrections.md#112-every-correction-records)). |
|
||
|
||
A textual body has a `value`, and optionally a `format` (`text/plain` or `text/markdown`) and a `language`.
|
||
|
||
The `source` of a `claim-link` or `symbol` body is a claim in a paperclaim claim store, written as paperclaim writes a reference: `claim:` and the claim's tag, eight Crockford base32 capitals (the digits and the letters without I, L, O and U), as in `claim:272XTX7C`, or `claim:<host>/<tag>` for a claim in another store, `<host>` being a domain name.
|
||
A tag never changes, while a claim's slug, such as `amber/k-bond`, may be renamed, so a slug is never a reference.
|
||
The optional `pin` fixes the claim as the annotator read it: the same reference, `@`, and the first six lowercase hex digits of the claim's statement hash, as in `claim:272XTX7C@02f47b`.
|
||
`common.schema.json` defines both, as `claimRef` and `claimPin`.
|
||
|
||
A `symbol` body names its symbol twice, as the target's span and by `key`, the symbol's key in the math form `scienting-math/1` ([9.4](09-figures-tables-equations.md#94-the-math-form)), which a claim store gives the symbol a symbol claim defines: `K_{r}` for a span `$K_r$`.
|
||
So a reader finds every occurrence of a symbol across a library by its key, however its source spaced or braced it, and a claim store matches the annotation to its claim without parsing the span.
|
||
The span is still required, since the key says which symbol and the span says where; `symbol`, optional, is the symbol as the span writes it.
|
||
|
||
A `comment`, `caveat` or `question` body MAY carry a `category`, which says what it is about, from a fixed list:
|
||
|
||
| `category` | Says |
|
||
| --- | --- |
|
||
| `comment` | A general remark. |
|
||
| `context` | Background, related work, or what a reader should know to follow the span. |
|
||
| `critique` | A flaw in the method, the analysis or the argument. |
|
||
| `question` | A question. |
|
||
| `correction` | An error of fact, number, citation or reference in the paper. |
|
||
| `feedback` | A suggestion addressed to the authors. |
|
||
| `resource` | Data, code or a related paper. |
|
||
|
||
A producer that supports categories SHOULD set one on every comment, caveat and question; the property is optional only so that layers written before it stay valid.
|
||
The category is independent of the kind, so a caveat may be a `correction` and a question a `critique`.
|
||
A `correction` category says that a reader found an error in the paper; correcting the document's reading of it is a correction of [11](11-corrections.md).
|
||
The store's `annotations` table keeps the category beside the body's kind ([13.4](13-database.md#134-the-store-tables)).
|
||
Nothing a model writes is ever stored as source text.
|
||
A model may propose a `description`, and like any annotation it stays `proposed` until a person confirms it; an accessible export uses confirmed descriptions only, and a figure with none has no description.
|
||
Other text a model writes about a span, such as a summary, is a `generated` annotation, which names its `generator`.
|
||
`fixtures/records/layer/` holds a layer with one annotation of each common kind, each comment, caveat and question with its category, a claim link and a symbol that reference a paperclaim claim and pin it, and the receipt that confirms some of them.
|
||
|
||
## 18.5 Traced citations
|
||
|
||
"As previously reported [12]": a reader who finds where the cited paper reports it marks the passage, and the `trace` body keeps the mark.
|
||
A `crosslink` says only that two spans correspond, in both directions, whereas a trace goes from one sentence's citation to one passage, and says which reference it follows, why, and who or what found it.
|
||
|
||
The annotation's target is a leaf or span anchor ([6.4](06-anchors.md#64-four-kinds-of-anchor)) on the citing sentence, which MUST be a leaf with runs, a sentence or a cell, and MUST cite the reference the body names: one of its `cite` runs lists `reference` ([3.5](03-nodes.md#35-inline-runs)).
|
||
`reference` is the ID of a `reference` node of the citing document, because a sentence may cite several references at once, as in `[3, 12]`, and the target says only which sentence.
|
||
`source` is the cited document, written as a target's source is, `tag:scient.ing,2026:doc:<store>/<key>`, and never the citing document itself.
|
||
`selector` selects the passage in it: an `AnchorSelector` whose anchor holds the passage's pin (its `node_hash`) and, for part of a leaf, its `span`, then a `TextQuoteSelector` when the anchor's quote is plain ([18.3](#183-the-annotation)).
|
||
The passage may be a container, such as a table, whose anchor has no quote.
|
||
|
||
| Property | Required | Holds |
|
||
| --- | --- | --- |
|
||
| `reference` | Yes | The reference node of the citing document that the sentence cites and the trace follows. |
|
||
| `source`, `selector` | Yes | The cited document and the passage in it. |
|
||
| `note` | No | What the passage reports, in a person's or a tool's words. |
|
||
| `verdict` | No | `supports`, `partly`, `does-not` or `cannot-tell`, as a `citation-context` body has it ([19.2](19-exercises.md#192-verdicts)). |
|
||
| `proposer` | When the creator is `Software` | `{model, run, score?, rank?}`: the model and run that found the passage as a candidate, as a `generated` body's `generator` names them, the candidate's `score` as a decimal string, and its `rank`, from 1, among the candidates the run proposed for this citation. |
|
||
|
||
The state of a trace is the annotation's `status`.
|
||
A trace a tool proposes is `proposed` until a person confirms it, and `rejected` when a person turns the candidate down; a rejected candidate is kept, so that the tool is not offered it again.
|
||
A trace is *disputed* while a reply to it, an annotation whose target is the trace and whose motivation is `assessing`, is `proposed` or `confirmed`.
|
||
The dispute is computed from the replies and never written on the trace, so a reply can be withdrawn.
|
||
|
||
Whether a trace still leads where it did is also computed, from the current state of both documents.
|
||
Resolve the target's anchor against the citing document and the body's anchor against the cited document ([6.5](06-anchors.md#65-resolution)).
|
||
The first row that applies gives the outcome:
|
||
|
||
| Outcome | When |
|
||
| --- | --- |
|
||
| `stale` | An anchor is stale, on the citing side, the cited side or both. |
|
||
| `unlinked` | The sentence no longer cites `reference`: a `relink` removed the citation ([11.7](11-corrections.md#117-citations-and-references)), or the reference was deleted. |
|
||
| `misresolved` | A `resolve` made the reference name a DOI or a document that is not the cited work. A resolution of `none` names nothing and contradicts nothing ([10](10-references.md)). |
|
||
| `changed` | Both anchors resolve and at least one is not `current`. |
|
||
| `current` | Both anchors are `current`, and the reference still resolves to the cited work or to nothing. |
|
||
|
||
A resolution names the cited work when its document is the cited document, another version of the same work (a key with the same work key, [2.4](02-model.md#24-work-and-document-keys)), or one the caller knows to be the work; and a `doi` resolution does when the work is known by that DOI (`same_as`, [2.4](02-model.md#24-work-and-document-keys)).
|
||
Besides the outcome, a resolution gives the passage's current address, `key#node@pin[start:end]`, while its anchor resolves, and the citations inside the passage.
|
||
A chain of "as previously reported" is then followed one trace at a time: the reader opens the passage, and a citation in it that has a trace of its own leads on.
|
||
|
||
A trace breaks `layer.trace.source` when its selector has no `AnchorSelector`, an anchor is in another document than `source` names, `source` is the citing document, or the `TextQuoteSelector` does not repeat the anchor's plain quote.
|
||
It breaks `layer.trace.target` when the target is not a leaf with runs of the citing document selected by an `AnchorSelector`, or the sentence does not cite `reference` ([22.4](22-conformance.md#224-rules)).
|
||
`vectors/traces.jsonl` resolves traces against the fixture packages after corrections, with every outcome and every refusal, and `fixtures/records/traces/` holds a layer of four annotations on `writer2027fixed-ynr7.v3` that cites `hale2026harmonic-damb`: a trace a model proposed to the paper's abstract and a person confirmed with a receipt, a candidate a person rejected, a person's trace to Table 1, and a reply that disputes the first.
|
||
|
||
## 18.6 Chemical structures
|
||
|
||
A paper writes about a molecule in words, and nothing in a converted document finds it there.
|
||
A `structure` annotation ties one to the words: its target is the span, as for any annotation, and its body holds the structure.
|
||
The words stay the paper's, the structure is one reader's reading of them, attributed and withdrawable, and a model's tie stays `proposed` until a person confirms it.
|
||
What a paper reports about a structure, such as the force-field constants of a bond, is about reading this paper ([18.1](#181-one-rule-decides-what-is-a-claim-and-what-is-a-layer)), so it is an annotation here; a claim store holds a parameter once it is checked.
|
||
|
||
| Property | Required | Holds |
|
||
| --- | --- | --- |
|
||
| `value` | Yes | The structure as a SMILES or a SMARTS, whichever `format` says. |
|
||
| `format` | Yes | `chemical/x-daylight-smiles` for a SMILES, `chemical/x-daylight-smarts` for a SMARTS. |
|
||
| `role` | Yes | What the span does with the structure: `names` it, `depicts` it (a figure, scheme or table draws it), `parametrizes` it (the span gives parameters for it) or `uses` it (a reagent, solvent or probe). |
|
||
| `identity` | Yes | What finds the same structure however it was written: the structure's InChIKey when it is a SMILES and the producer has one, else `smiles:` or `smarts:` and the first twelve hex digits of the SHA-256 of the value as written. |
|
||
| `written` | No | The structure as the paper writes it when that is not SMILES or SMARTS: `R-OH`, `CT-CT`. |
|
||
| `label` | No | A short name, such as `NMA`. |
|
||
| `accessions` | No | Entries of outside databases that the structure is: a list of `{db, id}`, with `db` in lower case (`pubchem`, `chebi`, `cas`). |
|
||
| `terms` | No | The bonds, angles, torsions and impropers the paper gives values for. |
|
||
|
||
A structure that is not one molecule, such as a group, a fragment with open positions or a pattern of atom types, is a SMARTS.
|
||
An open position is an atom that matches any atom, `[*:1]`, with its label as the atom map number, so `R-C(=O)-R'` is `[*:1]-C(=O)-[*:2]` and `R-OH` is `[*:1]-[OX2H]`, and the paper's own notation goes in `written`.
|
||
SMILES text that a toolkit would refuse, because a valence rule fails, is kept as a SMARTS rather than dropped, since a paper's drawing often is not a molecule a toolkit can sanitize.
|
||
|
||
**Atom numbers.**
|
||
Atoms are numbered from 0 in the order the value writes them, a bracket atom as one, so `CC(=O)NC` has atoms 0 to 4.
|
||
A hydrogen that is not written as an atom has no number: a term on a hydrogen needs `[H]` in the value.
|
||
The rule is of the text and not of any toolkit, which is what lets the numbers be checked without one.
|
||
|
||
A term has a `kind` (`bond`, `angle`, `torsion` or `improper`), `atoms` (2, 3, 4 and 4 atom numbers; an improper lists its central atom first), and optionally `types` (the force field's atom types, one for each atom), `values` (the parameters by the name the paper uses, each a decimal string as [1.2](01-conventions.md#12-numbers) writes numbers that carry meaning), `units` (a unit for each value) and `source` (where in the paper).
|
||
The atoms of a bond, angle or torsion are a chain, each bonded to the next, and the atoms of an improper are each bonded to the first.
|
||
|
||
A structure body breaks `layer.structure.value` when its value is not a SMILES or SMARTS as its format says: an unclosed bracket, branch or ring, a stray character, a bond with no atom, or a query (`,`, `;`, `&`, `!` or `$` inside a bracket, or `a` or `A`) in a SMILES.
|
||
It breaks `layer.structure.identity` when its identity is not of the form its value gives, or, for a `smiles:` or `smarts:` identity, is not the hash of the value.
|
||
It breaks `layer.structure.term` when a term cites an atom the value does not write or cites an atom twice, when its atoms are not joined as its kind needs, when `types` has another length than `atoms`, or when `units` names a value that is not given ([22.4](22-conformance.md#224-rules)).
|
||
The reference implementation, `paperspec.structures.problems`, reads atoms and bonds from the text with the standard library alone.
|
||
It does not check valence, aromaticity or the InChIKey, which need a chemistry toolkit; a producer with one SHOULD also check that the value sanitizes as a molecule when `format` says a SMILES, and that the InChIKey is the molecule's.
|
||
`vectors/structures.jsonl` reads the atoms and bonds of SMILES and SMARTS texts, mints identities, and breaks every rule, and `fixtures/records/chemistry/` holds a layer of four structures on `hale2026harmonic-damb`: ethane, named and confirmed; water, named by a model and still `proposed`; the CT–CT pattern of Table 1, a SMARTS with the bond constants of both parameter sets as terms; and water again, as a SMILES that writes its hydrogens, with the OW–HW constants.
|
||
The two writings of water share one InChIKey, which is what finds them as one.
|
||
|
||
## 18.7 Sequences
|
||
|
||
A `sequence` annotation ties a protein, DNA or RNA sequence to the words that name it, show it or report a variant of it, as [18.6](#186-chemical-structures) does for a structure.
|
||
|
||
| Property | Required | Holds |
|
||
| --- | --- | --- |
|
||
| `value` | Yes | The residues in capital letters, a gap as `-`. |
|
||
| `alphabet` | Yes | `protein`, `dna` or `rna`, which says what the letters mean. |
|
||
| `role` | Yes | `names` the sequence, `depicts` it (the span shows it, as text, a figure or an alignment), `uses` it (a construct, primer or probe) or `reports` something about it (a variant, a region). |
|
||
| `identity` | Yes | The alphabet, a colon and the first sixteen hex digits of the SHA-256 of the residues with the gaps taken out, so a sequence is found whatever gaps it was shown with and whatever its numbering. |
|
||
| `start` | No | The number of the first residue of the value in the molecule's own numbering; 1 when omitted. |
|
||
| `label` | No | A short name, such as `HP35`. |
|
||
| `organism` | No | The organism the sequence is from, as the paper names it. |
|
||
| `accessions` | No | Entries of outside databases that the sequence is, as in [18.6](#186-chemical-structures) (`uniprot`, `genbank`, `refseq`, `ensembl`, `pdb`). |
|
||
| `regions` | No | Stretches of the sequence the span is about: a list of `{start, end, label?}`, both ends included. |
|
||
| `variants` | No | Changes the span reports: a list of `{position, ref, alt}`. |
|
||
|
||
The alphabets allow the twenty amino acids, selenocysteine (`U`), pyrrolysine (`O`), the ambiguity codes `B`, `Z`, `J` and `X` and a stop (`*`) for `protein`; `A`, `C`, `G`, `T` and the IUPAC ambiguity codes for `dna`; the same with `U` for `T` for `rna`; and `-` for a gap in all three.
|
||
A residue's number is `start` plus the number of residues before it, gaps not counted, so a fragment of a protein is numbered as the whole protein is, and `regions` and `variants` use those numbers.
|
||
A variant says that the residue at `position` is `ref` in the value and `alt` in the variant; `alt` is a residue, `*` for a stop or `-` for a deletion.
|
||
|
||
A sequence body breaks `layer.sequence.value` when its letters are not those of its alphabet or are all gaps, `layer.sequence.identity` when its identity is not the hash of its residues, `layer.sequence.region` when a region starts after it ends or reaches outside the residues, and `layer.sequence.variant` when a variant is outside the residues, names a `ref` the value does not have at that position, changes a residue to itself, or has an `alt` its alphabet does not allow ([22.4](22-conformance.md#224-rules)).
|
||
The reference implementation is `paperspec.sequences.problems`.
|
||
It needs no toolkit, since an alphabet is a set of letters; a producer that knows the molecule SHOULD also check the sequence against the `accessions` it names.
|
||
`vectors/sequences.jsonl` mints identities and breaks every rule, and `fixtures/records/biology/` holds a layer of three annotations on the draft `writer2027fixed-ynr7.v3`, whose last sentence tests its charges on the villin headpiece subdomain HP35: the words that name it, the printed sequence with its three helices as regions, and a model's proposal of the N68H variant, in the numbering 42 to 76 that the headpiece uses.
|