feat: validator, bundle builder and release checks for 0.1.0
- Add tools/validate.py: validates packages, and with --correction runs proposed corrections through the gate against a package without writing anything (spec/11 §11.4). The gate now refuses a proposal that fails the correction schema, rule correction.schema. - tools/load.py validates every package before loading, and with --bundle --built-from builds a scienting-bundle/1 with the FTS5 search index and bundle_meta (spec/13 §13.5). Search rows are defined per unit (sentence, cell, equation outside a sentence, figure-text) with the parent kind as context. - Roster keys carry a role, and a key with the role owner may revoke any receipt (spec/08 §8.7, spec/20); roster schema, store table and the fixture roster updated. The checker signs a trial revocation to prove the rule both ways. - A qa: receipt raises the nodes it sampled except those in its errors (spec/08 §8.2); the example paper's fidelity counts become 27 F3, 13 F2, 4 F1 and 1 F0. - The destination chooses the keyed scheme, not the text policy: hmac-team between the team's machines, hkdf-pages outside the team (spec/05 §5.8, spec/12 §12.5). - A TextQuoteSelector must repeat the anchor's plain quote and is absent for hashed or keyed anchors (spec/18 §18.3). - Add fixtures/records/suspect.json, a scienting-suspect/1 record. - Oracle: hashes are kept incrementally and checked after every correction against a full recomputation; load and write refuse unsafe manifest paths; Ed25519, HKDF and schema validation come from the cryptography and jsonschema packages; canonical JSON uses the built-in encoder with an exact RFC 8785 fallback. - The tools need Python 3.14 with cryptography and jsonschema; the py310 CI environment is dropped and the workflow runs the default only. - Resolve every [CHECK] note in the docs, READMEs and context; add CONTRIBUTING.md; every tool answers --help and refuses unknown options. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
89 files changed
+1141
-513
No files matched your search
+32
-4
@@ -1,8 +1,12 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Load packages into a SQLite database with the tables of sql/doc.sql and sql/store.sql.
|
||||
|
||||
Non-normative tooling, standard library only.
|
||||
Non-normative tooling.
|
||||
Usage: python3 tools/load.py --sqlite lab.db PACKAGE [PACKAGE ...]
|
||||
python3 tools/load.py --sqlite lab.db --bundle --built-from REV PACKAGE [PACKAGE ...]
|
||||
With --bundle the new file is a scienting-bundle/1: it also gets the tables of sql/bundle-sqlite.sql,
|
||||
the full-text search index and bundle_meta (spec/13 §13.5).
|
||||
Each package is validated first, and nothing is loaded if any package fails.
|
||||
Files without tables of their own (assets, datasets, derived files) are copied by SHA-256 into an
|
||||
asset store folder beside the database, named <database>.assets/.
|
||||
"""
|
||||
@@ -15,22 +19,46 @@ import sys
|
||||
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
sys.path.insert(0, ROOT)
|
||||
|
||||
from tools.oracle import db # noqa: E402
|
||||
from tools.oracle import db, package # noqa: E402
|
||||
from tools.oracle.schema import Store # noqa: E402
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
ap.add_argument("--sqlite", required=True, help="the database file; created with the tables if new")
|
||||
ap.add_argument("--bundle", action="store_true", help="build a new scienting-bundle/1 file with search")
|
||||
ap.add_argument("--built-from", help="with --bundle: the store revision the packages come from")
|
||||
ap.add_argument("--store", help="with --bundle: the store's name (default: the packages' common store)")
|
||||
ap.add_argument("packages", nargs="+", help="package folders")
|
||||
a = ap.parse_args()
|
||||
if a.bundle and (os.path.exists(a.sqlite) or not a.built_from):
|
||||
ap.error("--bundle needs --built-from and a database file that does not exist yet")
|
||||
store, bad = Store(os.path.join(ROOT, "schemas")), False
|
||||
for p in a.packages:
|
||||
for e in package.validate(p, store).errors:
|
||||
print("%s: %s: %s: %s" % (p, e["where"], e["rule"], e["message"]), file=sys.stderr)
|
||||
bad = True
|
||||
if bad:
|
||||
sys.exit("Nothing was loaded.")
|
||||
new = not os.path.exists(a.sqlite)
|
||||
conn = sqlite3.connect(a.sqlite)
|
||||
if new:
|
||||
if a.bundle:
|
||||
db.create_bundle(conn, os.path.join(ROOT, "sql"))
|
||||
elif new:
|
||||
db.create(conn, os.path.join(ROOT, "sql"))
|
||||
for p in a.packages:
|
||||
key = db.load(conn, p, a.sqlite + ".assets")
|
||||
n = conn.execute("SELECT COUNT(*) FROM nodes WHERE doc_key = ?", (key,)).fetchone()[0]
|
||||
print("%s: %d nodes" % (key, n))
|
||||
if a.bundle:
|
||||
print("%s: %d nodes, %d search rows" % (key, n, db.index(conn, key)))
|
||||
else:
|
||||
print("%s: %d nodes" % (key, n))
|
||||
if a.bundle:
|
||||
stores = {r[0] for r in conn.execute("SELECT DISTINCT store FROM documents WHERE store IS NOT NULL")}
|
||||
store = a.store or (stores.pop() if len(stores) == 1 else None)
|
||||
if store is None:
|
||||
sys.exit("The packages name no single store; give one with --store.")
|
||||
db.write_meta(conn, store, a.built_from)
|
||||
conn.commit()
|
||||
|
||||
|
||||
|
||||
Reference in new issue
Block a user