feat: zipped packages, target set, checklists, audit fixes and a stress test
Features for the next six months (accessible HTML export, paperpack and
Lode on packages, the personal bundle):
- Zipped packages: the oracle writes and reads `.sct.zip` archives under
the rules of spec/12 §12.2; validate.py and load.py accept them; a
byte-identical zip of the tiny package joins the fixtures, and two
broken archives (traversal, symbolic link) join fixtures/invalid.
- The target set of targeted F3 (spec/08 §8.10) is computed by the
oracle and pinned by vectors/targets.jsonl under each digit rule; the
rule now covers cells as well as sentences and the leaves of anchored
containers, and defines a node's section.
- scienting-checklist/1 (provisional): a conformance check of an export,
covered by a receipt as ["checklist", id, hash] (spec/08 §8.11), with a
WCAG 2.1 AA fixture and its receipt.
- QA reports are signed like receipts (spec/08 §8.9).
- A `description` annotation body with role alt or long, so an
accessible export can find a confirmed figure description; the layer
fixture carries one, and the store's body-kind constraint allows it.
- scienting-spot-check/1, the seed payload, is registered with a schema.
- exercise, submission and grade are provisional until a course has
used them.
Audit fixes, each with a vector or a broken package where one was
missing:
- Folder packages: symbolic links are refused on validate and load,
manifest paths must not collide under case folding, listed files are
matched by exact name, files are hashed in chunks under a size limit.
- Files are compared byte for byte, so CR LF, a byte order mark, NaN,
a lone surrogate or invalid UTF-8 is refused instead of translated;
integers beyond 2^53 are read as doubles, as canonical JSON writes them.
- Receipts: signature.key must equal by and the key must be valid when
signed; a qa receipt's nodes must be the sample at the original's
hashes, its scope must name qa.of, its errors must be sampled nodes;
statuses are computed once, iteratively, so chains of spot-checks
hundreds deep settle; Base64 and manifest shape errors become refusals
instead of tracebacks; counts and the fidelity of roster-less packages
are recomputed.
- The gate enforces rule 11 of spec/11 §11.4: changed nodes are
validated against the node schema and the references between nodes
are checked, so a set to a bad value, an insert with unknown
properties or a delete of a referenced header cell is refused; a
relabel to null drops the label; a reclassify drops foreign
attributes; a split strips whitespace across runs; ids.jsonl order
follows spec/04 §4.4 in every state; a move after itself is refused.
- Anchors: the relabel rule requires that nothing else changed below the
node; an alias chain ending in a deletion no longer crashes resolve.
- Notation: a script after an empty group applies to the previous atom
({}^{14}C); letters are defined by Unicode block in spec/05 §5.4 and
the oracle; \left) is complex.
- Page fingerprints delete whitespace before folding, so "10 -20" and
"10-20" fold alike (spec/07 §7.1).
- Grades are summed with exact decimals; the reviewer error bound is
computed with integer arithmetic.
- receipts and revocations are keyed by (doc_key, id) in SQL, since one
revocation may be carried by several packages.
Checking:
- tools/stress.py: seeded randomized property tests over canonical JSON,
normalization, notation, random correction walks, anchor resolution,
Merkle roots, spot-check samples, error bounds, fingerprints, mutated
archives and packages, rights and spot-check chains; CI runs it.
- check.py verifies every count and version the prose states against
the repository, the zipped fixture and crafted archives, the checklist
coverage, and nine more broken packages.
- CONTRIBUTING.md gains a release checklist.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
142 files changed
+2312
-268
No files matched your search
+18
-15
@@ -3,18 +3,21 @@
|
||||
Non-normative; see spec/19-exercises.md.
|
||||
"""
|
||||
|
||||
from decimal import Decimal
|
||||
from fractions import Fraction
|
||||
|
||||
from . import anchors as A
|
||||
from .tree import Refusal
|
||||
|
||||
|
||||
def dec(fr):
|
||||
fr = Fraction(fr)
|
||||
if fr.denominator == 1:
|
||||
return str(fr.numerator)
|
||||
s = "%.4f" % float(fr)
|
||||
return s.rstrip("0").rstrip(".")
|
||||
def dec(d):
|
||||
"""A decimal string, exactly as the sum came out (spec/19 §19.4: points are decimal strings, summed exactly)."""
|
||||
return format(Decimal(d), "f")
|
||||
|
||||
|
||||
def _zero_like(d):
|
||||
"""Zero with the scale of `d`, so a missed element worth "0.5" scores "0.0"."""
|
||||
return Decimal(0).quantize(Decimal(d))
|
||||
|
||||
|
||||
def _resolved(state, anchor):
|
||||
@@ -57,34 +60,34 @@ def grade(body, submission, state):
|
||||
items = []
|
||||
|
||||
def item(name, pts, got, fb):
|
||||
items.append({"element": name, "points": dec(pts if got else 0), "max": dec(pts), "feedback": fb})
|
||||
items.append({"element": name, "points": dec(pts if got else _zero_like(pts)), "max": dec(pts), "feedback": fb})
|
||||
|
||||
if kind in ("anchor", "check"):
|
||||
if kind == "check":
|
||||
v = key["verdict"]
|
||||
ok = ans.get("verdict") == v["value"]
|
||||
item("verdict", Fraction(v["points"]), ok, v.get("feedback", {}).get("hit" if ok else "miss", ""))
|
||||
item("verdict", v["points"], ok, v.get("feedback", {}).get("hit" if ok else "miss", ""))
|
||||
for i, el in enumerate(key["elements"]):
|
||||
ok = _any(state, rule, ans.get("anchors", []), el["anchors"])
|
||||
item("element:%d" % i, Fraction(el["points"]), ok, el.get("feedback", {}).get("hit" if ok else "miss", ""))
|
||||
item("element:%d" % i, el["points"], ok, el.get("feedback", {}).get("hit" if ok else "miss", ""))
|
||||
elif kind == "trace":
|
||||
chain = ans.get("chain", [])
|
||||
for i, link in enumerate(key["chain"]):
|
||||
ok = i < len(chain) and _any(state, rule, chain[i], link["anchors"])
|
||||
item("link:%d" % i, Fraction(link["points"]), ok, link.get("feedback", {}).get("hit" if ok else "miss", ""))
|
||||
item("link:%d" % i, link["points"], ok, link.get("feedback", {}).get("hit" if ok else "miss", ""))
|
||||
elif kind == "reproduce":
|
||||
ok = abs(Fraction(ans["value"]) - Fraction(key["value"])) <= Fraction(key["tolerance"])
|
||||
item("value", Fraction(key["points"]), ok, key.get("feedback", {}).get("hit" if ok else "miss", ""))
|
||||
ok = abs(Decimal(ans["value"]) - Decimal(key["value"])) <= Decimal(key["tolerance"])
|
||||
item("value", key["points"], ok, key.get("feedback", {}).get("hit" if ok else "miss", ""))
|
||||
elif kind == "verify":
|
||||
flags = ans.get("flags", [])
|
||||
for i, err in enumerate(key["errors"]):
|
||||
hit = [f for f in flags if _any(state, rule, [f["anchor"]], err["anchors"])]
|
||||
ok = any(f["verdict"] == err["verdict"] for f in hit)
|
||||
fb = err.get("feedback", {}).get("hit" if ok else "miss", "")
|
||||
item("error:%d" % i, Fraction(err["points"]), ok, fb)
|
||||
item("error:%d" % i, err["points"], ok, fb)
|
||||
else:
|
||||
raise Refusal("grade.kind", "unknown exercise kind " + kind)
|
||||
score = sum(Fraction(x["points"]) for x in items)
|
||||
total = sum(Fraction(x["max"]) for x in items)
|
||||
score = sum((Decimal(x["points"]) for x in items), Decimal(0))
|
||||
total = sum((Decimal(x["max"]) for x in items), Decimal(0))
|
||||
return {"format": "scienting-grade/1", "submission": submission["id"], "exercise": submission["exercise"],
|
||||
"score": dec(score), "max": dec(total), "items": items}
|
||||
Reference in new issue
Block a user