alexmandClaude
ea0a7efa78
Features for the next six months (accessible HTML export, paperpack and
Lode on packages, the personal bundle):
- Zipped packages: the oracle writes and reads `.sct.zip` archives under
the rules of spec/12 §12.2; validate.py and load.py accept them; a
byte-identical zip of the tiny package joins the fixtures, and two
broken archives (traversal, symbolic link) join fixtures/invalid.
- The target set of targeted F3 (spec/08 §8.10) is computed by the
oracle and pinned by vectors/targets.jsonl under each digit rule; the
rule now covers cells as well as sentences and the leaves of anchored
containers, and defines a node's section.
- scienting-checklist/1 (provisional): a conformance check of an export,
covered by a receipt as ["checklist", id, hash] (spec/08 §8.11), with a
WCAG 2.1 AA fixture and its receipt.
- QA reports are signed like receipts (spec/08 §8.9).
- A `description` annotation body with role alt or long, so an
accessible export can find a confirmed figure description; the layer
fixture carries one, and the store's body-kind constraint allows it.
- scienting-spot-check/1, the seed payload, is registered with a schema.
- exercise, submission and grade are provisional until a course has
used them.
Audit fixes, each with a vector or a broken package where one was
missing:
- Folder packages: symbolic links are refused on validate and load,
manifest paths must not collide under case folding, listed files are
matched by exact name, files are hashed in chunks under a size limit.
- Files are compared byte for byte, so CR LF, a byte order mark, NaN,
a lone surrogate or invalid UTF-8 is refused instead of translated;
integers beyond 2^53 are read as doubles, as canonical JSON writes them.
- Receipts: signature.key must equal by and the key must be valid when
signed; a qa receipt's nodes must be the sample at the original's
hashes, its scope must name qa.of, its errors must be sampled nodes;
statuses are computed once, iteratively, so chains of spot-checks
hundreds deep settle; Base64 and manifest shape errors become refusals
instead of tracebacks; counts and the fidelity of roster-less packages
are recomputed.
- The gate enforces rule 11 of spec/11 §11.4: changed nodes are
validated against the node schema and the references between nodes
are checked, so a set to a bad value, an insert with unknown
properties or a delete of a referenced header cell is refused; a
relabel to null drops the label; a reclassify drops foreign
attributes; a split strips whitespace across runs; ids.jsonl order
follows spec/04 §4.4 in every state; a move after itself is refused.
- Anchors: the relabel rule requires that nothing else changed below the
node; an alias chain ending in a deletion no longer crashes resolve.
- Notation: a script after an empty group applies to the previous atom
({}^{14}C); letters are defined by Unicode block in spec/05 §5.4 and
the oracle; \left) is complex.
- Page fingerprints delete whitespace before folding, so "10 -20" and
"10-20" fold alike (spec/07 §7.1).
- Grades are summed with exact decimals; the reviewer error bound is
computed with integer arithmetic.
- receipts and revocations are keyed by (doc_key, id) in SQL, since one
revocation may be carried by several packages.
Checking:
- tools/stress.py: seeded randomized property tests over canonical JSON,
normalization, notation, random correction walks, anchor resolution,
Merkle roots, spot-check samples, error bounds, fingerprints, mutated
archives and packages, rights and spot-check chains; CI runs it.
- check.py verifies every count and version the prose states against
the repository, the zipped fixture and crafted archives, the checklist
coverage, and nine more broken packages.
- CONTRIBUTING.md gains a release checklist.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
44 lines
3.3 KiB
Markdown
44 lines
3.3 KiB
Markdown
# 19. Exercises, submissions and grades
|
|
|
|
Because every sentence and cell has a permanent address, the answer to a verification exercise is an address, and an address can be graded.
|
|
The three formats of this chapter, exercise, submission and grade, are `provisional` ([0.4](00-introduction.md#04-status)): no course has used them yet, and the first that does will shape them.
|
|
|
|
## 19.1 Exercises
|
|
|
|
An exercise is an annotation of kind `exercise` whose `exercise` property holds a `scienting-exercise/1`: an `exercise` kind, a `prompt`, a `key` and a `rule`.
|
|
|
|
| Kind | The student | The key |
|
|
| --- | --- | --- |
|
|
| `anchor` | Finds the sentence or cell that supports a statement. | `elements`: each a set of acceptable anchors, with points and feedback. |
|
|
| `check` | Decides whether a cited passage supports the citing sentence, and anchors the passage. | A `verdict` with points, and `elements`. |
|
|
| `trace` | Follows a value back to its original measurement. | A `chain`: an element per link, in order. |
|
|
| `reproduce` | Digitizes a plot and recomputes a value. | A `value`, a `tolerance`, a `unit`, the confirmed `digitization` it comes from, and points. |
|
|
| `verify` | Checks a model's `summary`, written with addresses, in which errors were planted. | `errors`: each a set of anchors and the correct verdict, with points. |
|
|
|
|
The rule's `match` is `same-node` (the student's anchor resolves to the node a key anchor resolves to), `overlap` (and, where the key has a span, the student's span covers at least `min_overlap` of it; default `0.5`), or `chain` (each link matches in order, with credit per link).
|
|
Feedback is `{hit, miss}` per element.
|
|
|
|
## 19.2 Verdicts
|
|
|
|
| Verdict | Means | As claim evidence, if a person confirms it |
|
|
| --- | --- | --- |
|
|
| `supports` | The passage says what the citing sentence claims. | `supports` |
|
|
| `partly` | It says some of it, or with a qualification the citing sentence drops. | `qualifies` |
|
|
| `does-not` | It does not say it, or says the opposite. | `disputes`, or no record when merely irrelevant; the reviewer chooses. |
|
|
| `cannot-tell` | The passage, or the student's access to it, is not enough. | No record. |
|
|
|
|
## 19.3 Submissions
|
|
|
|
A submission, `scienting-submission/1`, holds the exercise's ID, the document and revision, a pseudonymous `student` ID, and `answers`: `anchors`, a `verdict`, a `chain` of anchor lists, a `value`, or `flags` (each an anchor and a verdict), as the kind needs.
|
|
The student's name never enters a submission; the mapping from pseudonym to student stays in the course's records.
|
|
|
|
## 19.4 Grading
|
|
|
|
Before grading, every anchor in the key is resolved, and if any is stale nothing is graded with it.
|
|
Each element of the key earns its points when any of the student's anchors matches any of its anchors under the rule; a `check` earns its verdict's points when the verdicts are equal; a `reproduce` earns its points when the value is within the tolerance; a `verify` earns an error's points when a flag matches it with the correct verdict.
|
|
Points are decimal strings, summed exactly.
|
|
The result is a `scienting-grade/1`: the submission, the exercise, the `score`, the `max`, and an item per element with its points, its maximum and its feedback.
|
|
`grading.jsonl` grades a hit, a miss and an overlap.
|
|
|
|
Student submissions are educational records, so they live in the course's private store, never on a public log.
|