alexmandClaude 960cf6cbd3 feat(validate): 2.1 errors for the glossary registries
Validation now checks what 2.1 adds: registry ids carry their prefix and
are unique across all registries (E2101); a target's terms, equations,
values, and misconceptions name declared entries (E2102); equation
symbols resolve (E2103); values give a value or an ordered range and a
declared term (E2104); citation keys in the new registries resolve
(E2105); brace references in student-facing prose resolve (E2106);
`conflicts` and `target_text_forbid` patterns compile (E2107); and a bank
option's misconception_id is declared (E2108).

An unresolved {lo-...} or {t-...} in a field 2.0 already had (objective
and target text, reading notes) is left to the linter, because 2.0
accepted it and rendered it literally; the new reference kinds and the
new fields are checked everywhere.

Adds the `regex` crate, with default features off, for the course's own
patterns. Each error has a test built from the prototype's self-test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 11:58:53 -05:00
2026-10-17 12:02:11 -04:00
2026-08-07 15:48:11 -04:00
2026-08-07 15:48:11 -04:00
2026-08-10 01:52:15 -04:00
2026-08-07 15:48:11 -04:00
2026-08-07 15:48:11 -04:00
2026-08-07 15:48:11 -04:00
2026-11-02 08:22:07 -05:00
2026-08-07 15:48:11 -04:00
2026-11-02 08:22:07 -05:00
2026-08-07 15:48:11 -04:00

coursebank

coursebank runs the assessment side of a course as version-controlled data. Questions, exams, grading exports, and the statistics computed from them live as YAML (a plain-text data format) and Parquet (a binary table format) files in a git repository, so they can be reviewed in pull requests and tracked as each question is reused across terms.

It is both a command-line tool and a Rust library. The command line covers the whole loop: author items, validate and lint them, assemble an exam, export it for print or Canvas, ingest the grading export, analyze it, and write the results back so the next assembly knows how each question has behaved. The write-back step is narrower than it sounds: calibrate --apply writes the pooled statistics to analysis/calibration.yaml, and it never edits a bank file. A calibration: block inside a bank is a validation error.

This README describes what the code does. The code wins where an older guide, a doc comment, or the --help text says something different, and Known limits and unfinished parts lists the differences.

Contents

The problem and the design commitments

An exam is usually a document, and the information that makes it worth reusing is scattered: which questions discriminated between strong and weak students, which wrong answer attracted students who held a particular misconception, which question was used last term. The crate documentation in src/lib.rs states the premise directly: questions, exams, and grading exports are worth treating as a dataset rather than as a pile of documents. Treating them that way makes four routine tasks cheap: knowing which questions discriminate, catching a poorly worded item from its distractor pattern, telling a student which misconception their wrong answer indicates, and not reusing a question three terms in a row.

The design rests on a few commitments.

  • Records are the single source of truth for reuse: no separate ledger file exists. The reuse history of a question is derived by scanning every assessment record for its id. The records hold the answer to when a question was last used, and the answer cannot disagree with them.
  • A fingerprint covers only what a student saw: the fingerprint of an item is computed from its stem, its options (with which are correct), and its stimulus. Tags, topics, and other metadata do not contribute. Retagging an item keeps its pooled statistics valid, while rewording the stem changes the fingerprint and marks the statistics stale.
  • Validation reports everything at once: validate collects every problem and prints them together, so you fix a file in one pass instead of one error at a time.
  • Validation and linting are separate: validate finds problems that must be fixed. lint finds weaknesses in item craft, and every lint rule has a code that you can silence.
  • Small samples are labelled: an analysis of fewer than 100 examinees says so, and states the standard error it uses.
  • No database and no hidden state: a course is a directory in a git repository. The tool holds no state of its own, so a course set up in 2026 opens in 2031 with whatever version of coursebank you have then.

The files

A course is a directory of four kinds of file:

Path Holds
course.yaml course identity, policy, learning objectives, lectures
banks/*.yaml items, with their design intent
assessments/*.yaml what was given, in what order, on what date
data/*.parquet one row per student per item

The first three are hand-editable YAML meant to be read in a diff. The response data is machine-written and stored in Parquet, so pandas, polars, DuckDB, and R can read it without this tool. Writing comma-separated values (CSV) instead is available with ingest --format csv, and reading accepts either format regardless of which one wrote the file.

The pooled statistics for each item (difficulty, discrimination, per-option selection rates, flags, and any item response theory (IRT) parameters) live in a fifth file, analysis/calibration.yaml, written only by calibrate --apply. A command that loads the course attaches those statistics to each item by id, so the statistics are available wherever the item is used, though the bank file never contains them.

The files course.yaml, references.yaml, lectures/*.yaml, and objectives/*.yaml are one course file split by subject, and a course that keeps everything in course.yaml loads unchanged. The full directory layout is:

Path Role Written by Ignored by init's .gitignore
course.yaml identity, policy, units you no
references.yaml works cited you no
lectures/*.yaml one lecture each: readings and what it teaches you no
objectives/*.yaml one learning objective each, with its learning targets you no
banks/*.yaml item banks you no
assessments/*.yaml assessment records assemble, then you no
seals/*.yaml frozen copy of what was printed seal no
analysis/calibration.yaml pooled statistics per item calibrate --apply no (meant to be committed)
analysis/administrations/*.csv cohort aggregates for one administration analyze items --record no (meant to be committed)
data/*.parquet or *.csv one row per student per item ingest yes
reports/ per-student and cohort reports report yes
build/ generated exports export yes
schema/ JSON (JavaScript Object Notation) Schemas for editor validation init, schema no
templates/ Typst templates and typst.yaml template dump, template config no

coursebank course files lists the files the course is assembled from, coursebank course build prints the merged course, and coursebank course where ID says which file defines a given unit, lecture, objective, target, reference, or stimulus id. Every command merges the fragments in memory, and each id may be defined in only one place. course build is for reading and for other programs: nothing in coursebank reads what it writes.

coursebank init writes a .gitignore that excludes build/, reports/, data/, *.pdf, *.salt, .coursebank-salt, .DS_Store, and *.swp. init keeps an existing .gitignore and inserts only the missing patterns at the top, under the comment # Added by coursebank init. The directory split is deliberate: data/ and reports/ carry one row or file per student and stay out of git, while analysis/ holds cohort aggregates only and is meant to be committed.

Installing and building

coursebank builds from source. The sources do not record any other install route [CHECK: confirm there is no packaged install other than the nightly and release archives described under Versioning and releases]. The binary needs no runtime setup and reads no environment variables. The only environment variable in the crate is COURSEBANK_VERSION, read at build time: when it is set and non-blank, it replaces the version that --version prints, and otherwise the package version is used.

The Rust requirement is stated in two places, and they differ. Cargo.toml declares rust-version = "1.85" and edition 2024, so Rust 1.85 or newer is the minimum the crate asks for. pixi.toml pins the toolchain to rust >=1.96.0,<1.97, so a pixi environment builds with a newer compiler than the minimum.

The repository uses pixi to pin the toolchain and wrap the common tasks:

pixi run build     # release binary at target/release/coursebank
pixi run install   # install the binary onto your PATH
pixi run tests
pixi run check     # verify: formatting, lint, tests, and doc build

pixi task list prints every task with a one-line description. pixi run install runs cargo install --path . --locked. pixi run cb validate -C path/to/course runs the command-line tool through cargo without installing it. pixi run check runs the format check, clippy with warnings as errors, the full test suite, the documentation examples, a check that every code fence in docs/ declares its language, and the rustdoc build.

Plain cargo works too with a recent toolchain:

cargo build --release

Two tools that coursebank does not bundle are needed to turn some exports into finished documents. coursebank writes Typst source (.typ) but does not compile it, so a Typst compiler must be installed separately. The pixi docs environment adds typst, and the hint that export typst prints is pixi run -e docs typst compile <file>.typ. The bundled Typst templates import the packages @preview/mitex:0.2.7 and, for student reports, @preview/whalogen:0.3.0 from the Typst package registry [CHECK: whether the compiler downloads these at compile time or finds them cached was not checked]. Quarto renders the .qmd files from export practice and export site, and the partials from lecture readings and lecture objectives are written for a Quarto site. export qti needs nothing beyond the binary.

The Cargo manifest defines no feature flags. The help text for export site, the pixi run build-lean task, and several guide pages describe parquet and site features. No [features] table exists in Cargo.toml, and no source file gates code on a feature. Parquet storage and export site are always compiled in, and instructions such as cargo build --features site refer to a feature that the manifest does not define.

Quick start

Create a course, validate it, then assemble and export an exam from the pool:

coursebank init --code "BIOSC 1540" --title "Computational Biology" --term 2026s
coursebank validate
coursebank assemble exam-4 --levels 1=6,2=8,3=10,4=6 --forms 2
coursebank export typst exam-4 --form A

init refuses to overwrite an existing course.yaml. It writes a skeleton course.yaml (one unit, one lecture, one objective, and one target as placeholders to rename), the JSON Schemas in schema/, and the .gitignore described above. assemble draws from the pool of approved items, so on a new course with no approved items it fails with a message that no approved items match the blueprint.

Run coursebank --help for the full command set.

The loop, end to end

The loop is: author items, validate and lint them, assemble an exam, export it, seal it before printing, give the exam, ingest the grading export, analyze it, and calibrate so that the statistics are stored for the next assembly. The sequence below is built from the commands in the guide pages and in the two course repositories, in the order a real exam follows [CHECK: none of these commands was run for this README; the flags come from the code and the course scripts, so run the sequence on a scratch course before relying on it]. The assessment id exam-2 and the objective ids are examples.

# 1. Author: create an empty bank, then write items in banks/sequence-analysis.yaml
coursebank bank new sequence-analysis --title "Sequence analysis"

# 2. Check: validate must pass, lint is advice
coursebank validate
coursebank lint
coursebank catalog --coverage

# 3. Assemble: draw two forms from the approved pool
coursebank assemble exam-2 --title "Exam 2" --kind exam --date 2026-03-24 \
  --platform paper --levels 1=2,2=1,3=2,4=1 --bonus 5=1 \
  --require lo-align-scoring=2,lo-dock-scoring=1 \
  --max-per-bank 4 --cooldown 180 --forms 2 --seed 20260324
coursebank assessment show exam-2

# 4. Export: Typst source for each form, then compile it with typst
coursebank export typst exam-2 --form all

# 5. Seal: freeze what is about to be printed, before printing
coursebank seal exam-2

# 6. Give the exam and grade it in Gradescope, one export directory per form.

# 7. Ingest: names are read but not stored; pseudonymize with a salt kept outside the repo
coursebank ingest gradescope A=grading/exam-2/form-a B=grading/exam-2/form-b \
  --assessment exam-2 --pseudonymize --salt-file path/outside/the/repo/salt --dry-run
coursebank ingest gradescope A=grading/exam-2/form-a B=grading/exam-2/form-b \
  --assessment exam-2 --pseudonymize --salt-file path/outside/the/repo/salt
coursebank data

# 8. Analyze: look first, decide any drop or regrade, then record the aggregates
coursebank analyze items exam-2
coursebank analyze items exam-2 --record
coursebank analyze irt exam-2
coursebank analyze students exam-2
coursebank report cohort exam-2 --typst
coursebank report students exam-2 --typst

# 9. Calibrate once, after every administration is ingested, then lint again
coursebank calibrate
coursebank calibrate --apply
coursebank lint

A few orderings matter.

  • Seal after export and before printing: the seal records the printed letters and the digests of what was printed. Sealing after ingest records drift as if it were the truth.
  • Decide drops and regrades before --record: analyze items --record writes two administration files that are never rewritten, and it refuses to overwrite them. Edit the assessment record first (dropped: true, dropped_as: full_credit, or credit_overrides) when a question must be dropped or regraded, and do not edit scores by hand.
  • Calibrate once, after the loop: calibrate pools every stored administration, so running it after each exam recomputes the same pool and running it before the last ingest leaves that exam out.
  • Lint again after calibrating: five lint rules compare the observed statistics with the author's predictions in the design block, and they have nothing to compare until calibrate --apply has written analysis/calibration.yaml.
  • Pass the assessment id as an argument: the guide pages show forms such as analyze items --assessment exam-2; the command line takes the id as a positional argument, and calibrate takes no assessment at all.

Authoring items

What an item is

An item is a question plus two things a question does not normally carry: what you predicted about it before anyone answered, and what happened when they did. The prediction lives in the item (the design block). The outcome lives in analysis/calibration.yaml, keyed by item id.

An item has these fields (unknown keys are errors, so a misspelled field is caught):

  • id, status, level, stem (required), and format (default single_best_answer).
  • cognitive_process, title, stimulus, points, bonus, topics, prerequisites, author, notes_private.
  • options, each with id, text, correct, credit, explanation, hint, misconception, error_type, feedback_student, selection_rate_expected, defensible, defense, and retired.
  • learning_targets (alias learning_objectives), the targets the item is tagged to.
  • sources, each naming a lecture with optional slides, readings, and recording_seconds.
  • solution, with model_answer, explanation, rubric, accepted, and review citations.
  • assets, each with path, alt, and caption.
  • design, with expected_difficulty (0 to 1), expected_discrimination (low, moderate, or high), expected_time_seconds, and rationale.
  • review (reviewed_by, reviewed_on, action, notes) and a retired block (on, reason, replaced_by).
  • supersedes, for an item that replaces another.

Three conventions differ from the older guide pages. Statistics are never in a bank. The version: and history: fields on an item are ignored and never written, and migrate stems removes them. Items are tagged with learning_targets, and the older name learning_objectives is accepted as an alias.

Options form a pool. An item may hold several defensible keys and more distractors than any form shows, and assemble draws which are shown. Option ids are names of the form o- followed by lowercase letters, digits, and hyphens (for example o-fourth-line), and the letters A to H from earlier versions are still accepted. An option id identifies an option; it is not its printed position, because shuffled forms relabel options on output.

Levels, formats, and status

Levels follow the revised Bloom taxonomy and are written as integers: 1 Remember, 2 Understand, 3 Apply, 4 Analyze, and 5 Evaluate or Create. A cognitive_process must belong to the item's level; the code defines 19 values:

Level Cognitive processes
1 recognize, recall
2 interpret, exemplify, classify, summarize, infer, compare, explain
3 execute, implement
4 differentiate, organize, attribute
5 check, critique, generate, plan, produce

The four formats are single_best_answer, multiple_response, true_false, and open_response. An open_response item has no options and is not machine-scored. Status is one of draft, in_review, needs_revision, approved, and retired, and only an approved item without a retired block is eligible for assembly. An approved item must also have a cognitive_process, at least one learning target, at least one source, and a design block, and an approved open_response item needs a solution with a model_answer or a rubric.

The error_type of a distractor names the kind of mistake it was written to catch. The 13 values are recall_confusion, terminology_swap, misconception, incomplete_understanding, overgeneralization, plausible_irrelevant, dropped_step, reversed_relationship, wrong_procedure, sign_or_magnitude_error, ignores_interacting_condition, correct_but_incomplete, and common_shortcut. The text a student sees for a chosen option comes from feedback_student first, then misconception, then explanation. That ordering allows a student report to tell a student which misconception their wrong answer indicates.

Partial credit on an option needs a defense and defensible: true, and it must respect the course policy (allow_partial_credit, and partial_credit_floor_level).

Writing stems and options

Stems and options use a subset of Typst with some Markdown conveniences: *bold*, _italic_, backticks for inline code, #sub[x] and #sup[x], and a fixed table of #sym. symbols such as #sym.arrow.r, #sym.times, and #sym.degree. Math goes between dollar signs as LaTeX and is handed to the Typst package mitex. Chemistry written as \ce{...} goes to the whalogen package, which the template must import, and the exporter warns if it does not. The markup is converted per target: HTML for QTI (Question and Test Interoperability) and the site, Unicode for Markdown and plain text, and Typst for printed documents. The conversion is not identical everywhere. A single-asterisk *x* is emphasis in HTML and QTI. In Typst output it is bold.

Course-level structure

Learning objectives and learning targets form two tiers. An objective is what a report classifies, and a target is what items are tagged to. An id may exist in only one of the two registries, and an item that tags an objective that has targets is a validation error: tag the target. Lectures list the objectives they teach, and readings attach to lectures with a ref, a locator, a role (assigned or supplemental), and the targets they support. The policy block carries defaults such as points_per_item (1.0), options_per_item (4), mastery_threshold (0.75), min_items_for_mastery (2), allow_partial_credit (true), bonus_levels, and a grade_scale.

Editor support

init and schema write eight JSON Schemas into schema/: course, references, lecture, objective, calibration, measurements, bank, and assessment. Put a modeline at the top of a YAML file and an editor with a YAML language server validates the file as you type:

# yaml-language-server: $schema=../schema/bank.schema.json

The schemas are written by hand for autocompletion, and the crate's own validation remains the authority. The schema command prints a modeline that points at ../.coursebank/schema, which is not where the schemas are written. Use the ../schema/ form shown above.

What validate checks, and what lint checks

coursebank validate checks that every file is correct and consistent, and it exits 2 if it finds any problem. It covers the course (empty identity fields, grade scale, duplicate ids, reference fields, prerequisite cycles, level ceilings), every bank against the course (the rules above, unknown targets, lectures, stimuli, and references), every assessment record against the catalog, every seal's stem digests, and calibration against the banks. A bank item that carries a calibration: block fails validation.

coursebank lint checks craft. It has 30 rules in five families, and each rule has a code.

Family Codes What they flag
Cueing (9) cue-key-longest, cue-uneven-length, cue-word-repeat, cue-all-of-the-above, cue-absolute, cue-overlap, cue-near-duplicate, cue-key-position, cue-option-count features that let a test-taker guess without knowing
Clarity (6) clarity-no-task, clarity-stem-length, clarity-negation, clarity-vague, clarity-reading-load, clarity-unfocused stems that do not pose a clear task
Completeness (7) complete-distractor-design, complete-distractor-explanation, complete-student-feedback, complete-alt-text, complete-format-level, complete-bonus-policy, complete-design-conflict missing design intent, feedback, or alt text
Pool (3) pool-thin, pool-unused, pool-unjustified-retirement too few usable distractors for a form
Evidence (5) evidence-nonfunctioning, evidence-stale, evidence-difficulty, evidence-discrimination, evidence-duplicate-stem observed results that contradict the prediction or the text

Each finding has a severity of low, medium, or high, and a finding's severity can differ from its rule's nominal one (for example, cue-uneven-length is medium when the spread is extreme). Retired items are skipped, and findings are grouped by item. The options are --list-rules (list every code and severity, then exit), --only CODES, --ignore CODES (both comma-separated), --min-severity low|medium|high (default low), and --no-fail. lint exits 2 if it finds anything, unless --no-fail is given, which is why a pre-commit hook of coursebank validate && coursebank lint works. To block a commit only on serious findings, add --min-severity high.

coursebank catalog summarizes the pool: item, bank, and assemblable counts, and counts by status and by level. catalog --coverage adds a table per objective and per target (items, ready items, targets, maximum level) and lists gaps, exiting 2 if there are any. The gap kinds are an uncovered objective, an objective with no approved items, an objective with a single assemblable item, an objective covered only at level 1 despite a higher ceiling, a lecture with no assessed objective, and an item with no objective.

Assembling an exam

assemble draws a new assessment from the pool and writes it to assessments/ID.yaml. It takes a blueprint on the command line:

Flag Meaning Default
--title TITLE assessment title the id
--kind exam, quiz, homework, practice, or final exam
--date YYYY-MM-DD the date, which also anchors the cooldown today
--platform paper, canvas, or other paper
--levels L=N,... scored items per level, for example 1=6,2=8,3=10,4=6 none; assemble fails if no scored items are requested
--bonus L=N,... bonus items per level none
--require OBJECTIVE=N,... at least N items for an objective, satisfied by items on any of its targets none
--lectures, --topics, --banks restrict the draw (comma-separated; the filters intersect) no restriction
--max-per-bank N at most N items from any one bank no limit
--cooldown DAYS avoid items used within this many days 180
--seed N seed for the draw 0
--forms N number of forms 1
--dry-run show the draw, write nothing off
--force overwrite an existing record off

assemble fails with a message if no scored items are requested. It has no --term flag: the record takes the course term. It refuses to overwrite an existing record unless you pass --force, because replacing an administered assessment invalidates stored responses.

How the draw works

  1. The pool: only approved, unretired items are eligible, narrowed by the bank, lecture (an item's sources), and topic filters. An empty pool is an error.
  2. Objective minimums first: for each --require entry, assemble picks that many non-bonus items tagged to the objective or to one of its targets, respecting the per-bank cap and the cooldown. A shortfall is a note, not a failure.
  3. Level quotas: for each level in --levels, in ascending order, it ranks the remaining candidates and fills the quota. The first pass honors the cooldown and the per-bank cap. A second pass relaxes the cooldown when the first falls short, and adds a note such as "reused ... inside the 180-day cooldown". assemble fails when items are still short, and says how many are blocked by the cooldown and by the cap.
  4. Bonus: the same fill runs over items flagged bonus: true.
  5. Ordering: scored items are sorted by level and then id so that the form ramps in difficulty, and bonus items are numbered after the scored ones.

Items placed for an objective minimum are not counted against the level quotas, so the scored total can exceed the sum of the --levels counts.

The ranking is deterministic. Candidates are shuffled with a generator seeded from the seed and a label, then stable-sorted so that never-used items come first, then the item whose last use is oldest, then the item with the fewest uses. The shuffle is the tie-break. No system entropy is involved, so the same seed, the same pool, and the same history give the same exam.

Two changes illustrate the effect. Lowering --cooldown from 180 to 30 stops the first pass from blocking items last used between 30 and 180 days ago, so assemble is less likely to need the second pass or to fail with a count of items blocked by the cooldown. The ranking still prefers never-used items. Changing --seed changes the tie-breaks among equally fresh items and leaves the freshness ordering alone.

What the record stores

For each placement the record stores the question number, the item id, the stem digest, the option-set variant digest, the fingerprint, the points, whether it is a bonus item, the key (option ids keyed), the distractors (option ids offered), the level, and the learning targets. It also stores the course term, the blueprint, and the forms. The options shown are drawn once, by assemble, and written to the record. They are never sampled at export, so every export of the same record shows the same options. The draw does not fail when an item's pool is too short for a full form; the form is short and lint reports pool-thin.

Placement number is the printed question number and the join key to grading exports. Numbers must run 1 to N without gaps. Option order on paper is computed from the form's seed and is never stored, so the paper, the key, and the answer sheet agree. assemble writes forms with shuffle_items: false and shuffle_options: true, and gives each form a seed derived from "{assessment id}/form-{label}". Form labels run A, B, and on to Z, then AA. To give a form a different arrangement, edit the seed: in the record, re-export, and re-seal with --force (allowed only for an assessment that was never administered).

Corrections made after the fact go in the record: credit_overrides (a map from option to a credit between 0 and 1) and dropped: true with dropped_as: removed or full_credit. dropped and credit_overrides cannot be combined on one placement.

Inspecting a record

assessment new ID writes an empty record (default --kind exam) for a hand-built assessment, and validate reports that it lists no items until you add some. assessment list shows ID, kind, date, item count, and points for every record. assessment show ID prints the by-level counts and, for every question, its number, id, key, points, and estimated working minutes, then validates the record and exits 2 if there are problems.

The working-minutes estimate uses these per-level values when an item has no design.expected_time_seconds:

Level Seconds
level 1 35
level 2 65
level 3 95
level 4 130
level 5 165

An assessment is found by its file name or by the id: inside the record. In both course repositories the ids are the id: values (A1.2, e1), while the file names are lowercase with hyphens (a-1-2.yaml), and everything coursebank does is keyed on the id.

Reuse history

Reuse is derived from the records and never stored. usage history lists, for every item that has been used, when and where (as assessment#number) and the last-used date. Name an item (usage history ITEM) to see one. usage unused lists approved items that no record has ever placed. assemble reads the same history, so the cooldown and the preference for never-used items rest on the records you can read in a diff. The history spans every assessment record, of any kind and any term. Deleting an assessment record removes its uses from the history, so keep records that were administered.

Sealing before printing

seal ID freezes an administration: what the students held in their hands. The seal stores a copy of each stem and option text as administered, the printed-letter-to-option map for every form and every question, and SHA-256 digests per item, per form, and for the whole file. It writes seals/ID.yaml. The purpose is practical. A paper exam shuffles options, so a Gradescope export names letters as printed on that form, and only the seal can translate a printed C on form B back to the option it was. Ingest derives the map from the record as it stands today and warns when no seal exists, and that map is wrong if the bank or record has changed since printing.

Flag Effect
--check compare the existing seal with the live course, print findings with severity, and exit 2 on drift
--force overwrite an existing seal; use it only for an assessment that was never administered
--form A,B seal only these forms (default: every declared form)
--no-content store fingerprints and digests only, not stem and option text
--out PATH write somewhere other than seals/ID.yaml

A seal is meant to be written once, before the exam is printed. seal --check reports drift with a code and a severity, and only high severity blocks. High-severity codes include a changed whole-file or form digest, a removed, replaced, or edited item, a changed key, a changed option count, a removed form, and a changed option permutation. Medium covers changed points and an added item, and low covers retagged objectives and a changed level. The command prints per-form counts and advisories about key balance after writing a seal.

A seal also records dropped questions (the e1 seal of one course repository lists dropped: [31, 35]), and when migrate counters rewrites an item id, the old file digest is appended to superseded_digests. validate checks every seal's stem digests against the bank, so a reworded stem is caught even long after the exam.

The digests come in three kinds. An item's fingerprint is a hash of its trimmed stem, each option (sorted by id) as id|correct|text, and the stimulus id, using FNV-1a, which is not a security primitive. The stem digest covers the stem and stimulus only, and the variant digest covers the stem digest plus the options as administered and which were keyed; the variant digest is the key under which statistics are pooled. Seals use SHA-256 and prefix digests with sha256:.

Canvas ingest does not use the seal, because a Canvas export names answer text and not printed letters, and the homework in the two course repositories has no seals.

Exporting

Every export reads the assessment record and the bank, so a re-export of the same record gives the same documents.

Typst: paper exam, key, and answer sheet

export typst ID writes Typst source, one file per form and variant, into build/ by default:

  • ID-FORM.typ (the exam), ID-FORM-key.typ, and ID-FORM-answer-sheet.typ (a bubble sheet sized from each item's option count).
  • --form takes one form id (default A) or all. The option takes a single value, whatever the help text's "repeat or pass all" suggests.
  • --variant selects documents. The default set is exam, key, answer-sheet.
  • --out DIR changes the directory, --json also writes the payload as JSON, and --dry-run prints the question count and the resolved template and writes nothing.
  • --template FILE is accepted only when exactly one --variant is given.

The parser also accepts the tokens student-report and cohort-report. Those documents are built from responses, not from an assessment record, so export typst has nothing to build them from [CHECK: the code comment says a diagnostic cannot be built from an assessment record, but what export typst --variant student-report actually does was not run]. Use report students --typst and report cohort --typst for them.

The tool writes Typst source and does not compile it. Compile with typst compile (in the pixi docs environment, pixi run -e docs typst compile FILE.typ). export typst exits 2 when it printed an advisory warning: an unbalanced [ or ] in a stem or option, a template with no markers, or a chemical formula in a slot when the template lacks a whalogen import.

You own the layout. In the words of docs/TYPST.md, the tool does not write Typst documents: it loads a Typst file you own, finds the markers in it, and injects data. The markers are line comments, // coursebank:begin SLOT and // coursebank:end SLOT, and the text between them is replaced. The markers define three slots: meta (a #let cb-meta = (...) binding), questions (one #render-question((...)) call per printed item), and data (a #let cb-data = (...) binding with the metadata and the questions). The marker lines survive, so an exported document can be reused as a template. Malformed markers (an unknown slot, an unclosed region, a stray end, a nested region, a slot claimed twice) are all reported at once.

The template is found in this order: a --template path or template: in the render config, then templates/ID-VARIANT.typ, then templates/VARIANT.typ, then the built-in template compiled into the binary. The five built-in templates are exam.typ, key.typ, answer-sheet.typ, student-report.typ, and cohort-report.typ. template list shows the lookup chain, which entry resolved, and the slots each template declares. template dump writes the built-in templates into templates/ to edit (all five by default, --variant to choose, --force to overwrite, --stdout to print), and it exits 2 if it skipped an existing file. init does not create templates/; template dump does.

templates/typst.yaml configures the payload in three layers: built-in defaults for the variant, then defaults:, then variants:. template config writes a starter file and template config --resolved VARIANT prints the fully resolved settings. The keys include question_fn, content (content or str), letters (upper, lower, numeric, roman, nothing), reveal, stimulus, fields, number_from_record, and a free-form extra. Unknown keys are rejected.

reveal decides whether the payload contains the answer at all. Its values are nothing, key, and everything, and for the exam the correct-answer field is absent from the payload. The built-in exam and student-report variants use nothing, and the built-in key and cohort-report variants use everything [CHECK: the sheets do not say what key includes that nothing leaves out, or what everything adds to key; confirm against docs/TYPST.md]. Do not set reveal: key on the exam to make a solutions copy: export the key variant instead. Because option order comes from the form's recorded seed and question numbers come from the record, the paper, key, and answer sheet always agree. The full template contract is in docs/TYPST.md.

QTI: Canvas

export qti ID builds a QTI 1.2 package for Canvas in the common-cartridge style, written to build/ID-FORM.zip. Import it in Canvas under Settings, Import Course Content, QTI .zip file. The flags are --form (default A), --out FILE, --no-feedback (leave out per-option feedback), and --attempts N. The item types map as follows.

Item format QTI type
single_best_answer multiple_choice_question
multiple_response multiple_answers_question (full credit only for the exact set)
true_false true_false_question
open_response essay_question

The essay mapping has not been round-tripped through a live Canvas import in this build, according to a comment in the code. A choice item with no keyed option aborts the export and lists the problems. Ids are derived by hashing the assessment id and item id, so a re-export keeps identical ids. Inline math is rewritten to \( ... \) because Canvas's MathJax does not read bare dollar signs. The attempt limit, the scoring policy, and shuffling come from the assessment record (attempts, default -1 for unlimited), not from the command line. The command accepts --attempts and ignores it. Figures from item assets are not included in the package.

Practice worksheet and solutions

export practice ID writes Quarto Markdown: ID-FORM.qmd (stems and options only) and ID-FORM-solutions.qmd (the key, worked reasoning, per-option notes, the rubric, and review citations). --variant worksheet|solutions selects one (both by default), --out DIR sets the directory (default build/), and --no-answer-space drops the written-answer space after open-response questions. Render with quarto render FILE.qmd.

Site: gated solutions for a web page

export site ID writes a Quarto partial and an encrypted solutions bundle for a password-gated homework page. It writes _questions.qmd and ID-solutions.json to --out (default build/). The partial contains the questions and an empty hidden slot per question, with no answers in the file; include it from a page with {{< include _questions.qmd >}}. --assets DIR also writes questions.css and solutions.js, which a Quarto site adds through _quarto.yml. In the browser, solutions.js derives the key from the password, decrypts every solution, and re-typesets the math. It needs a secure context, because it uses the Web Crypto application programming interface (API).

The bundle is encrypted with a key derived by PBKDF2-HMAC-SHA256 (250,000 iterations), and each solution is sealed with AES-256-GCM under its own random 96-bit initialization vector. The command generates a password unless you pass --password. A generated password is 16 characters from the alphabet 0123456789abcdefghjkmnpqrstvwxyz, shown in groups of four (80 bits), from the system random generator. The command prints the password and stores it nowhere, so write it down when it appears. The guide describes the lock as a deterrent, not a grading system of record (docs/guide/assignment.md).

Markdown review copy

export md ID writes the items as Markdown for a reviewer, to build/ID.md. --with-key adds correct-option marks, the rationale, and the option explanations.

Lecture and reference outputs

lecture readings ID and lecture objectives ID write the readings block and the learning-objectives block for one lecture, as Quarto Pandoc definition lists (--style quarto, the default) or as plain Markdown bullets (--style plain). They print to standard output without --out. references export writes the bibliography as hayagriva (the default, read by Typst), csl-json (read by Zotero and Pandoc), or bibtex. The label, role, base_url, and note fields do not survive into those formats.

Ingesting grading data

Two grading exports are supported, and the assessment id is required for both.

Command Reads
ingest gradescope SOURCE... one directory of per-question CSVs per form, files named 1.csv to N.csv, or N_Title.csv for a titled question
ingest canvas FILE the Canvas "Student Analysis" quiz CSV (not the gradebook, not the Item Analysis)

Both readers produce the same long-format row, one per student per item. No other platform is implemented, even though the platform field can say paper, canvas, or other.

The flags shared by both commands are:

Flag Meaning
--assessment ID the assessment record the responses belong to (required)
--date YYYY-MM-DD the administration date (default: the record's date)
--form ID which form, when forms were used
--pseudonymize replace student identifiers with keyed pseudonyms; requires --salt-file
--salt-file PATH the secret for the keyed hash; at least 16 bytes once whitespace is stripped
--format parquet|csv storage format (default Parquet)
--dry-run parse and report, write nothing

ingest gradescope adds --allow-mismatch and --recorded-numbers.

Several forms

Write each Gradescope source as FORM=DIR, for example A=exports/e1-a B=exports/e1-b. A bare path takes its form from --form, and giving neither is an error. A form given twice, or a form the record does not declare, is an error. Gradescope numbers questions by printed position, and the decoder maps each printed position to the recorded question number and each printed letter to the option it was, using the seal. The decoded rows keep the printed letters in selected and eliminated and add the bank-lettering in selected_source and eliminated_source.

Three checks guard against a mislabelled directory or a bad key. The first compares each directory's graded key with every declared form's printed key, so that swapped directories are detected; a mismatch blocks the ingest unless you pass --allow-mismatch, which turns it into a warning. The second flags a student who appears on more than one form, and a form that is missing a question the record expects. The third warns when two forms differ in mean score by 5 percentage points or more with at least 5 students on each. --recorded-numbers is for an export that already uses recorded question numbers. Ingest also warns when a distractor received partial credit at grading time, and the analysis then flags the item as ambiguous. Ingest uses a seal when one exists and prints any drift as warnings.

What the readers do

The Gradescope reader takes the keyed options from the rubric columns whose point value equals the top value, treats true in a rubric cell as a selection, and computes correct as whether the chosen set equals the keyed set. A blank row has correct empty. response_time_seconds is always empty for Gradescope. The Canvas reader works from answer text, because Canvas exports text and not letters. It recovers the letter by matching normalized answer text against the item's option text, and an answer that does not match keeps its score, loses its letter, and triggers a warning. It aligns questions to item numbers by matching stem text against the record, falling back to record order with a warning. Canvas ingest does not use the seal, so there is no printed-letter translation.

Privacy

The tool reads student names from the export and does not store them: names are not part of the stored schema, so reports fall back to the student key. Without --pseudonymize, the stored rows still carry the institutional id, email, and section. With --pseudonymize, the student key becomes s- followed by 12 hexadecimal characters of an HMAC-SHA-256 of the lowercased, trimmed id (the institutional id, else the email, else the existing key), keyed with the salt, and the sid, name, and email are cleared. coursebank refuses --pseudonymize without a salt file, so a file never looks anonymous when it is not. Keep the salt outside every repository. The error message suggests generating it with openssl rand -hex 32 > PATH, where PATH is a location outside the repo. According to the course scripts, losing the salt exposes nobody; it only ends the ability to match a student across administrations. init's .gitignore excludes *.salt and .coursebank-salt, and data/ is ignored as a whole.

What gets stored

Ingest prints the number of responses, students, and items and writes one file per administration into data/. The administration id is COURSE/TERM/ASSESSMENT, using the course code exactly as written, and the file name is that id sanitized: lowercased, with letters, digits, ., and - kept and every other run replaced by _. For example, with the course code BIOSC 1540, the administration BIOSC 1540/2026s/exam-4 is stored as biosc_1540_2026s_exam-4.parquet. Writing replaces an existing file for that administration and does not append, so re-ingesting after a correction is safe. coursebank data lists what is stored, with rows, students, items, and file.

The Parquet file is Snappy-compressed, with every column declared non-nullable and an empty string or zero used where a value is unknown. The columns, in order, are:

  • identity and context (text): administration_id, course, term, assessment_id, date, form
  • student (text): student_key, sid, email, section
  • the item: item_number (the recorded question number and the join key to the record), item_ref (the item id), item_version, variant (the option-set digest as administered)
  • the response: selected, eliminated (comma-joined letters as printed on that student's form), correct (1, 0, or empty), credit (fraction of the item's points), points_possible, score, response_time_seconds
  • item metadata copied from the record and bank: level, learning_objectives (comma-joined target ids; the column keeps its old name), topics, bonus, dropped
  • the decoded form: form_position, dropped_full_credit, selected_source, eliminated_source

On read, variant, form_position, dropped_full_credit, selected_source, and eliminated_source are optional, so older files still load. A file with other missing or mistyped columns is rejected with a message saying it was probably not written by coursebank. Reading accepts either Parquet or CSV regardless of the format that wrote the file.

Analysis and calibration

Classical test theory

analyze items ID runs a classical item analysis on one administration. --pooled reads every stored administration of that assessment id across terms, and without it only the current term is read. Only scored, undropped, non-bonus items enter the analysis. For each question it reports:

  • Difficulty, p: the mean of the 0/1 correct column over all examinees, counting a blank as incorrect. The blank rate is reported separately. Partial credit is coded as incorrect for these statistics.
  • Discrimination, r: the corrected point-biserial, the Pearson correlation between the item's 0/1 column and the total score on the other items. It is undefined when either column has no variance.
  • Discrimination index, D: the proportion correct in the top group minus the proportion correct in the bottom group, where each group is the top or bottom 27% of students by total score. It is omitted for fewer than 6 students or when the groups overlap.
  • Distractor analysis: for each option, the selection rate, the option's point-biserial, and its rate in the upper and lower groups. Every declared option appears even if nobody chose it.
  • Reliability: KR-20, the standard error of measurement, the mean p and the mean point-biserial, with a plain-language reading of alpha in bands from very low to very high (0.7 to 0.8 is read as acceptable for a classroom exam) and a note when the exam has fewer than 20 items.
  • Flags and the revise queue, described next.

A sample smaller than 100 examinees triggers a small-sample warning that quotes a standard error of 1/sqrt(n-1).

The analysis defines 11 flags. The table shows the defaults for the flags that use a threshold.

Flag Fires when
negative_discrimination point-biserial below -0.05
low_discrimination point-biserial below 0.15 (and not already negative)
too_easy p above 0.95
too_hard p below 0.25 and the point-biserial is not at least 0.15
distractor_outperforms_key a distractor's point-biserial exceeds the key's and it is chosen by at least 10%
key_underperforms a distractor's upper-group rate exceeds the key's and is at least 0.25
nonfunctioning_distractor a distractor is chosen by 5% or fewer
ambiguous a non-key option earned partial credit at grading
design_mismatch observed p is more than 0.25 from the predicted difficulty, or the point-biserial is outside the predicted band, on a calibration with at least 50 examinees whose fingerprint matches the item
high_rapid_guess see the limits section; cannot fire today
dif_flagged see the limits section; cannot fire today

Four flags block: negative_discrimination, distractor_outperforms_key, ambiguous, and key_underperforms. The revise queue lists flagged items with blocking flags first, then by lowest point-biserial, and analyze items prints the worst ten and exits 2 if the queue is not empty. A script can branch on that exit code to review the revise queue.

The bands for design.expected_discrimination are:

Band Point-biserial
low below 0.20
moderate 0.15 to 0.40
high 0.30 or above

--record also writes the aggregates under analysis/administrations/, described below.

Item response theory

analyze irt ID fits a one-parameter, two-parameter, or three-parameter logistic model by marginal maximum likelihood. --model takes rasch, two-pl (the default), or three-pl. The fit uses 41 quadrature points over a standard normal ability range of -4 to 4, and it estimates a discrimination a, a difficulty b, and (for 3PL) a guessing parameter c per item, with standard errors, plus an expected-a-posteriori ability for each examinee. Priors are on by default (a log-normal with a log standard deviation of 0.5, b normal with a standard deviation of 2), because unpenalized maximum likelihood has no finite solution for an item that everyone answered correctly. --no-priors turns them off and warns that divergence is likely on a single class. Warnings appear when n is below 100 for the Rasch model, 200 for the two-parameter model (2PL), and 1000 for the three-parameter model (3PL). Responses are dichotomous, so partial credit counts as incorrect.

Per-student mastery

analyze students ID reports a roster, the objectives the class did not meet, and archetype patterns from a seeded k-means (k = 3) over per-level rates, which runs only with at least 9 students and two levels. A student's mastery of an objective is Meeting when the rate is at least mastery_threshold, Developing when it is at least 0.6 of the threshold, NotYet otherwise, and NotEnoughEvidence when fewer than min_items_for_mastery items bear on the objective. Objectives are classified; targets are reported as itemized evidence. A rate is marked confident when the lower bound of its 95% Wilson interval clears the threshold, so the same rate over more items can be confident when it is not over fewer.

Recording an administration

analyze items ID --record writes two CSV files under analysis/administrations/, named for the slug of the administration id:

  • ...-items.csv: one row per question (administration id, assessment, term, date, forms, number of examinees, the coursebank version, item, number, variant, stem digest, n, p, point-biserial, discrimination index).
  • ...-options.csv: one row per question and option (item, number, option, keyed, selection rate, point-biserial, upper and lower group rates).

They are cohort aggregates, and the readers reject any header outside a fixed list, so a student column cannot slip in. They are meant to be committed, and they are written once and never overwritten. They carry no IRT parameters.

Calibration

calibrate pools the statistics from every stored administration and writes them back for the next assembly. It takes no assessment argument.

Flag Effect
--apply write analysis/calibration.yaml; without it, print the plan and write nothing
--no-irt skip the IRT fit
--include-practice include practice assessments (excluded by default)
--min-n N minimum pooled examinees before an item is written (default 10)

Each administration is analyzed on its own, so the point-biserial is measured against that administration's test, and the results are pooled per item. The pooled difficulty is the examinee-weighted average. The point-biserial and the discrimination index are examinee-weighted averages of the within-administration values, not recomputed from merged data. Flags are the union. Statistics are also kept per option set (the variant digest), so a stem that appeared with different options produces separate variants records. The plan prints a diff of what would change for each item, and tags items as new or as computed from different text. calibrate --apply rewrites only analysis/calibration.yaml, and each entry stores the fingerprint it was computed against. Every pass recomputes from all stored administrations, rather than accumulating deltas.

Pooled IRT is narrower than the other statistics. The IRT fit uses a single administration, the one with the most students-by-items cells, and it needs at least --min-n students and 5 items in it. A warning says that the parameters come from one administration, because linking across administrations is not attempted. Read irt in the calibration file as describing that one administration.

Where results are stored

Where What Git
data/ per-student responses not committed
analysis/ cohort aggregates: calibration.yaml and administrations/*.csv committed
reports/ per-student and cohort reports not committed (per-student files carry student keys)

Two cases that show how the pieces interact

Two cases show how changing one thing changes the outcome.

  • Retagging versus rewording: move an item from one learning target to another and its fingerprint is unchanged, so its pooled statistics stay valid and evidence-stale does not fire. Reword its stem and the fingerprint changes, the next calibrate plan tags the item as computed from different text, and evidence-stale fires at high severity until it is recalibrated. In an assessment record that already uses the item, a reworded stem is a validation error: give the new wording a new id and record supersedes.
  • Prediction, observation, and sample size: the design block predicts a difficulty and a discrimination band. evidence-difficulty fires when the observed difficulty is 0.25 or more from the prediction, and evidence-discrimination fires when the observed point-biserial falls outside the predicted band. The analysis flag design_mismatch adds a sample-size condition: it needs a calibration of at least 50 examinees whose fingerprint matches the item's current fingerprint, and below that a miss is recorded in the prediction notes only. The old calibration no longer matches the fingerprint after a rewording, so it cannot raise the flag until the item is recalibrated.

Reports

report students ID writes a diagnostic for each student, and report cohort ID writes an instructor item analysis. Both default to Markdown, and each can also write HTML (--html), Typst source (--typst, to compile with typst compile), and the payload as JSON (--json). --no-markdown suppresses the Markdown, and --template FILE uses your own Typst template.

The student report answers "what should I do next?" It includes the student's performance by level and by objective, strengths, the objectives to focus on, per-question results with the feedback and hint written for the option the student chose, dropped questions, lectures to review with readings, and a study list. It leaves out the correct answers, other students' data, and any numeric rank. Two optional switches change that: --solutions adds the worked solution for every missed question (the command warns about the effect on reusing the exam), and --misconceptions names the misconception each chosen distractor was written to catch, which is off by default. --ability adds the IRT ability estimate and --no-comparison omits the comparison with the class. Section switches turn individual sections off: --no-questions, --no-feedback, --no-hints, --no-levels, --no-objectives, --no-strengths, --no-focus, --no-dropped-questions, --no-review-lectures, and --no-study. Files go to reports/ID/, named for the student key, unless --out DIR says otherwise. The report is routed by the key, because the stored schema has no names. A pseudonymized ingest produces reports that cannot be routed to a student, which is why the two course repositories ingest with ids first, compile the reports, and then re-ingest pseudonymized (see the worked pictures).

The cohort report answers "what should I fix?" It holds the score distribution (mean, median, standard deviation, minimum, maximum, and ten bins), reliability, per-level and per-objective rows, objective gaps below the mastery threshold, letter-grade bins when policy.grade_scale is set, per-lecture and per-question rows, dropped-question detail, a triage for each flagged question, a comparison of predictions with results, per-form rows, and warnings. The triage buckets are discard (the point-biserial is below -0.05, or below 0.05 with p near chance), rekey (a distractor chosen by at least 20% with a positive point-biserial at least equal to the key's), bounded (a weak question that is too easy or too hard), revise (weak, or with a nonfunctioning option, or ambiguous), and reteach (a question with a point-biserial of at least 0.20 whose difficulty p is below the course mastery threshold). The default path is reports/ID-cohort.md, and the command exits 2 when the revise list is not empty. report students prints a drift warning to the error stream if the bank has moved since the seal.

Lectures, references, and site content

lecture coverage [--lecture ID] lists the readings behind each learning target and which targets have none. It exits 2 when an assessed target has no reading. references list prints every work with its kind, role, label, and link, and counts works with no link. A reference link is resolved in this order: an explicit url, then base_url plus path, then the reference's url, then the digital object identifier (DOI), arXiv id, PubMed Central (PMC) id, and PubMed id. course build merges the split course, and the course scripts of both repositories use its output to list lecture ids; the author's authoring pipelines read the same output as their course file.

Migrating an older course

Every migrate subcommand takes --dry-run, which shows what would change and writes nothing. The current schema version is 2.0, and major versions 1 and 2 are readable, so a 1.0 course loads unchanged and migration is optional.

Command What it does
migrate split split one course.yaml into references.yaml, lectures/, and objectives/; keeps the original as course.yaml.bak; reassembles and compares before reporting success (exits 2 if the reassembly differs)
migrate counters drop the trailing counter from every item id (q-x-001 becomes q-x) across banks, records, seals, and the response store; colliding ids abort
migrate order replace order: integers with ordered declarations; run migrate split first
migrate references turn citations written into note: fields into real fields (journal, volume, pages, DOI); never overwrites a declared field
migrate variants fill the stored variant column in data/ from the assessment records
migrate stems drop the ignored version: and history: fields
migrate options rewrite option letters as names derived from option text; run with --dry-run first; seals keep their letters on purpose; exits 2 if some items need hand-naming
migrate ids rewrite pre-2.0 bank::item ids as plain item ids in records, seals, and the response store

Run --dry-run first, and commit before applying. The hints that some migrate commands print afterwards name commands that do not exist (see the limits section).

Command reference

coursebank has 21 top-level names and 47 runnable commands: 9 single commands and 12 groups holding 38 subcommands. Every command accepts --course PATH (or -C PATH) and --quiet (or -q). Identifiers are positional.

Stage Command What it does
Project setup init --code C --title T --term TERM [--with-examples] create a course; refuses to overwrite course.yaml
schema write the eight JSON Schemas to schema/
validate check every file for problems that must be fixed; exits 2 on any
lint [flags] check item craft; see Authoring items
catalog [--coverage] [--topics] summarize the pool and objective coverage
course files, course build [--out PATH], course where ID list the source files, print the merged course, find where an id is defined
references list, references export [--format F] [--out PATH] list or export the bibliography
Authoring bank new ID [--title T], bank list create an empty bank, list banks
assessment new ID [--title T] [--kind K], assessment list, assessment show ID create, list, or inspect a record
usage history [ITEM], usage unused reuse history; approved items never used
Assembly assemble ID [flags] draw a new assessment from the pool
Sealing seal ID [--check] [--force] [--form A,B] [--no-content] [--out PATH] freeze what is printed; check for drift
Export export qti, export typst, export md, export practice, export site see Exporting
template list, template dump, template config manage Typst templates
Ingest ingest gradescope SOURCE..., ingest canvas FILE read a grading export
Analysis analyze items ID, analyze irt ID, analyze students ID analyze one assessment
calibrate pool every stored administration into analysis/calibration.yaml
report students ID, report cohort ID per-student and instructor reports
data list the stored administrations
Lectures lecture readings ID, lecture objectives ID, lecture coverage site partials; reading coverage
Migration migrate split, counters, order, references, variants, stems, options, ids upgrade an older course; all take --dry-run

coursebank has no import, guide, or completions subcommand. The only help is --help, and the guide pages are rustdoc modules.

Conventions that apply to every command

  • Course location: --course PATH or -C PATH names the directory that holds course.yaml (default: the current directory). A command that loads the course fails with a usage error if there is no course.yaml, and the message says to run init or to use --course.
  • Version: --version prints the build-time version string.
  • Quiet: --quiet or -q suppresses some trailing hints and advisories.
  • Exit codes: 0 is success, 1 is a failure (printed to standard error as error: ...), and 2 means the command ran and found problems. Exit 2 comes from validate (any problem), lint (any finding, unless --no-fail), catalog --coverage (gaps), lecture coverage (an assessed target without a reading), assessment show (problems in the record), analyze items (a non-empty revise queue), report cohort (a non-empty revise list), seal --check (drift), export typst (any template warning), template dump (a skipped file), migrate split (reassembly differs), and migrate options (items needing hand-naming). Code 2 is why a course repository can have a pre-commit hook.
  • Unknown keys are errors: a misspelled field in a YAML file fails validation instead of being ignored.
  • Dates: every date is written YYYY-MM-DD; anything else is an error.
  • Count flags: options such as --levels and --bonus take LEVEL=COUNT with a level from 1 to 5; --require takes OBJECTIVE=COUNT.
  • Assessment lookup: an assessment is found by file name (assessments/ID.yaml) or, failing that, by the id: inside any record. A form is found by its id, case-insensitively, and a record that declares no forms gets an implicit form A.
  • No runtime environment variables: only the build-time COURSEBANK_VERSION exists. The pseudonymization salt is passed as a file, not an environment variable.
  • Determinism: no command uses system entropy for the draw. The one exception is the site export, which uses the system random generator for passwords and initialization vectors.

Using coursebank as a Rust library

The package builds two targets: the binary coursebank from src/main.rs and the library coursebank from src/lib.rs. The crate forbids unsafe code and warns on undocumented public items. The public modules are:

Module Contents
model the data model and validation: assessment, bank, calibration, catalog, course, history, item, layout, seal, taxonomy
authoring lint, select (the assembly draw), jsonschema
data canvas, decode, gradescope, intake, responses, store, store_parquet
analysis calibrate, classical, diagnostic, irt, students
export lecture, practice, qti, references, report, site, typst
util citation, date, hash, markup, rng, yaml, zipfile
error, migrate, guide the error type, migrations, and the guide pages

The crate root re-exports these submodules and the main types: Catalog, CourseFile, SCHEMA_VERSION, Error, Result, History, Item, Layout, and from the taxonomy CognitiveProcess, ErrorType, Flag, Format, Level, and Status. coursebank::VERSION is the package version, not the nightly string that --version can show.

The commands call these library modules, so a program can do the same work: load a course, assemble an exam, and analyze stored responses. The calls below come from the guide pages, where each is a doctest that is compiled without being run [CHECK: this block combines fragments from three guide pages and was not compiled as a whole; imports, error handling, and the construction of blueprint are left out, and return types such as Result may differ].

// Load a course and every bank, with calibration attached to each item.
let catalog = Catalog::load(Path::new("path/to/course"));

// Draw items. History comes from the assessment records.
let history = History::load(&catalog.layout.assessments());
let selection = select::select(&catalog, &blueprint, &history, Date::new(2026, 3, 24)?);

// Analyze a stored administration.
let store = Store::open("data");
let responses = store.read_assessment("exam-2");
let analysis = classical::analyze(&responses, &Thresholds::default(), None, None);
let queue = analysis.revise_queue();

The rustdoc guide modules, in order, are coursebank::guide::setup, authoring, first_exam, typst_export, assignment, and recipes. The Rust examples in them are doctests, most of them marked no_run, so pixi run tests compiles them. pixi run doc builds the API documentation and opens it, and pixi run doc-build builds it without opening a browser.

Worked picture: biosc1000

biosc1000 is a biochemistry course (term 2026f) whose repository holds two things: course/, the coursebank data, and website/, a Quarto site.

Shape and scale

  • Course directory: course.yaml (a root fragment with identity, policy, units, and one stimulus), 17 lecture files, 82 objective files, and references.yaml.
  • Merged course: full-course.yaml has 4 units, 17 lectures, 82 learning objectives, 399 learning targets, and 12 references, and is git-ignored.
  • Banks: 13 banks with 339 items (311 approved, 16 in review, 12 draft), 336 of them single best answer and 3 multiple response.
  • Assessment records: 15, made up of 13 homework sets and 2 exams.
  • Statistics: no bank item carries observed statistics; the banks hold only the authored predictions in design. The statistics are in analysis/calibration.yaml, which has 70 calibrated items, 34 from the first exam and 36 from the second, with 2PL parameters on 36 of them.
  • Committed: four administration CSVs (an items file and an options file per exam) are committed.

How the tool is called

The binary is a 16 MB file at course/coursebank, git-ignored and not on PATH, and it reports version 0.0.0. Scripts reach it through the variable COURSEBANK, whose default is ./coursebank. The repository has no wrapper of its own that calls it. Instead, a set of shared course scripts, installed once per machine as commands, do the calling. They read a per-course course.env (as data, never executed) and an exams.tsv manifest.

Script What it calls
course-merge course build --out to write the merged course
lectures-export course build to list lectures, then lecture objectives and lecture readings for each, into the website's _objectives.qmd and _readings.qmd
assessments-render export site, export typst, export qti, and export practice per assessment
exams-seal seal for each id in SEAL_IDS
exams-process validate, seal --check, ingest, analyze items, report, calibrate, lint

Across all of these, the scripts use 15 distinct command forms: course build, seal (and seal --check), export site, export typst, export qti, export practice, lecture objectives, lecture readings, validate, ingest (Canvas and Gradescope), analyze items (and --record), report cohort, report students, calibrate (and --apply), and lint [CHECK: the figure of 15 comes from the source sheet; the count depends on whether the flag variants are counted separately].

Homework to the site and Canvas, exams to paper

assessments-render chooses the formats from the kind of assessment. Homework gets a site page (export site), Typst, and a QTI zip for Canvas. Exams get Typst only, because they are on paper. Recitations would get the site and practice exports. The 13 homework sets each have a generated _questions.qmd and an encrypted ID-solutions.json under website/assignments/. The partials carry a banner saying they are generated and must not be edited, and lectures-export --check fails if they are out of date. The site exporter does not write the question level, so the script injects a level label from the record, with a comment to retire that once the exporter does it. Page passwords are sticky: a rerun reuses the stored password, and a key rotates only when OVERWRITE=true is set. The script also skips a page whose input hash (the record, all banks, course.yaml, and the binary) is unchanged.

An exam through the loop

Each exam follows the order that the scripts encode.

  1. Author the bank and the assessment record. The exams have two forms each (A and B, with shuffle_items: false and shuffle_options: true), 37 items each, and 75 minutes. The first exam's blueprint is 6, 18, 9, and 3 items at levels 1 to 4 plus one level-5 bonus item.
  2. exams-seal writes seals/ID.yaml before printing. Homework on Canvas has no seal.
  3. Print from the Typst files in build/.
  4. Grade in Gradescope. The manifest exams.tsv lists, for each exam, one directory per form (A=DIR B=DIR) and the flags --date and --format csv; a directory means a Gradescope export and a single file means a Canvas Student Analysis CSV, detected from the path [CHECK: the biosc1000 sheet could not confirm that the Gradescope ingest was actually run; it infers it from the manifest and the stored CSVs].
  5. exams-process runs per assessment: validate (skipped by default), seal --check, ingest ... --dry-run and then the real ingest, analyze items (look), a pause for "any question to drop or regrade?", and analyze items --record. It then compiles the cohort and per-student reports with Typst.
  6. The scrub step: the first ingest keeps ids so that the reports can be routed. The script re-ingests with --pseudonymize --salt-file after the reports are compiled, so that what stays in data/ has no identifiers. The script refuses a salt that lives inside the repository.
  7. Calibration runs once, after the loop: calibrate, then (after a confirmation) calibrate --apply, then validate, and finally lint, which runs after calibration because five lint rules compare evidence with predictions.

Stored data is CSV: the manifest passes --format csv, and the data directory holds two CSV files, one per exam, with one row per examinee and item (66 by 37 and 65 by 37) and no Parquet file. The first exam's seal records two dropped questions.

Where the outputs go

The scripts write to the three destinations in Where results are stored: data/, analysis/, and reports/. The repository's README also states that student data, the salt, answer-key PDFs, and build output are never committed.

The authoring pipelines around coursebank

The banks are written by a separate multiple-choice authoring pipeline that drafts a module bank (the module exam and the homework on the same lectures, planned together) and moves each item's status from draft to approved. Its last stage runs coursebank validate, and when the binary is not installed in a sandbox, the pipeline falls back to its own lint and records that coursebank validate is still owed. A separate lecture pipeline reads the coursebank course build output as its course file. Those pipelines are not part of coursebank.

The record for one homework (A3.2) was written by hand, and the author's notes keep the equivalent assemble command, which also shows --require in use.

coursebank assemble A3.2 --title "A3.2 - Enzyme kinetics" --kind homework \
  --date 2026-10-20 --platform canvas --levels 1=2,2=9,3=3,4=1 --bonus 5=1 \
  --require lo-rate-regimes=4,lo-kinetic-parameters=5,lo-catalytic-efficiency=2,lo-kinetic-data=5 \
  --lectures L3.2 --seed 3202 --forms 1

Worked picture: biosc1540

biosc1540 is a computational biology course (term 2026f) with the same shape as biosc1000, built on older copies of the scripts that live inside the repository, and the differences show what else the tool handles.

Shape and scale

  • Course directory: course.yaml (a 42-line identity, policy, and units file), 14 lecture files, 70 objective files (69 objectives and 356 targets), and references.yaml with 64 entries.
  • Banks: 11 banks with 284 items (238 approved, 45 retired, 1 draft), all single best answer.
  • Assessment records: 13 (11 homework, 2 exams).
  • Item design: every item has a design block, every option has feedback_student, 854 of the 1,138 options name a misconception and an error_type, and no bank item carries observed statistics.
  • Statistics: analysis/calibration.yaml has 72 items, 36 from each exam, and 36 of them have 2PL IRT with priors.

The scripts inside the repository

scripts/ holds 10 helper scripts, of which five call coursebank:

  • merge-course.sh runs course build --out full-course.yaml.
  • export-lectures.sh reads lecture ids from the course build output and runs lecture objectives and lecture readings for each, with --check and --prune options.
  • seal-exams.sh runs seal for each exam.
  • render-assessments.sh runs the four exports, with the same format defaults as in biosc1000.
  • process-exams.sh runs the whole post-exam loop from exams.tsv, and has the same scrub step and the same refusal of a salt inside the repository, and it hard-codes the binary as ./coursebank.

Each script changes into course/ first and calls ./coursebank or $COURSEBANK. render-assessments.sh includes the binary in its change hash, so replacing the binary makes every page re-render, while the stored passwords are reused.

What is different here

  • Migrated from the 1.0 layout: a commit titled "migrate to new coursebank" rewrote course.yaml and split lectures and objectives into their own files. A mix of schema versions remains: 5 banks and 6 assessments are at schema 1.0, and 6 banks and 7 assessments are at 2.0, and course.yaml still declares "1.0". The schema text in the course mentions coursebank migrate order. The mix works because both major versions are readable.
  • Multi-form sealed exams: the first exam declares three forms (A, B, and C, with shuffle_items: false and shuffle_options: true) and the second two (A and B) [CHECK: the manifest row for each exam lists two form directories, so confirm whether the first exam's third form was graded separately]. Both exams are sealed so that printed letters map back to option ids at ingest.
  • Dropped items and bonus items: the second exam has four items marked dropped: true with dropped_as: full_credit, which the stored data carries in its dropped and dropped_full_credit columns. Both exams use bonus items (a level 3 and a level 5 bonus on the first, a level 5 bonus on the second), and several homework sets mark bonus items.
  • Gradescope ingest stored as CSV: the manifest gives one directory per form and --format csv, so the two stores in data/ are CSV files with no Parquet file. The analysis records say 27 examinees for each administration.
  • Calibration with IRT: calibrate and calibrate --apply produced the 72-item calibration file.
  • The salt outside the repository: the salt's default location is a folder outside the repo, the COURSEBANK_SALT variable overrides it, and the script refuses a path inside the repo.
  • Reports: a cohort PDF and a per-student PDF (27 per exam) are produced with Typst, using the student flags --ability --no-comparison --no-study --no-focus --no-strengths. The reports are git-ignored.
  • No custom templates: the repository has no templates/ directory and no typst.yaml, so the built-in templates are used.
  • Recitation practice is wired and unexercised: render-assessments.sh has the export practice path (worksheet and solutions, rendered to PDF by Quarto). No recitation record exists, so that path has not been run in this repository.
  • Homework to the site: all 11 homework sets have a generated page with a stored password. The site keys are tracked in git although the repository's own notes say to never commit them.

What the two repositories teach about adopting the tool

The two repositories are the only evidence of the tool in use, and they agree on several habits.

  • Call the tool through shell scripts: the binary is a git-ignored file at course/coursebank, not on PATH, and the scripts change into the course directory, so ./coursebank resolves without -C.
  • The binary is not pinned: every seal and administration record in both repositories says the tool version is 0.0.0, because no release has been cut. Nothing in either repository records how the binary was built or which commit it came from. Rebuild deliberately, and expect that an older seal or record was written by a different build than the one you have now. [CHECK: the sheets do not say how the author obtains or updates these binaries, or whether they were built from this repository's current head.]
  • Keep the merged full-course.yaml out of git: it is derived by course build and rebuilt after editing fragments. Tools that need lectures read it, because course.yaml is only a root fragment.
  • Assessment ids come from id:: file names are lowercase with hyphens, the ids are A1.2 and e1, and every script, manifest row, and seal is keyed on the id.
  • Generated partials say "do not edit": the site files that coursebank writes are regenerated, and a --check mode fails if they are stale.
  • Keep student data out of git: student data, salts, reports, and answer-key PDFs are never committed. The salt lives outside every repository, and a scrub step removes identifiers from what stays on disk.
  • Passwords for site solutions are sticky: a rerun reuses the stored password so a released password keeps working, and rotation is deliberate. Both repositories track their password files in git, against their own scripts' warning not to commit them on a public repository [CHECK: the brief mentioned one repository, but both source sheets report tracked password files; confirm].

Three decisions are yours to weigh, and the repositories give no clean answer to any of them. The first is storage format. Both repositories pass --format csv, and the repositories do not record why. CSV opens in any editor, while Parquet is the default, is Snappy-compressed, and keeps column types. The second is whether to commit site passwords. The render script refuses to touch an existing solutions bundle when the stored _password.txt is missing, so losing the file strands a released password; committing the file prevents that and allows anyone who can read the repository to open the solutions. The third is when to calibrate. Running calibrate after each exam gives a running estimate, and running it once after the last exam gives a single pass, because it recomputes from every stored administration. The code labels any analysis with fewer than 100 examinees a small sample, and each administration in biosc1540 recorded 27, so weigh single-class statistics accordingly.

Known limits and unfinished parts

Each item below states what the code does.

Build and features

  • The feature flags parquet and site appear in help text, the lean-build task, and the guides. Cargo.toml defines neither. Parquet storage and export site are always compiled in.
  • Cargo.lock is git-ignored, although pixi run install uses --locked.
  • The Rust version is declared as 1.85 in Cargo.toml and pinned to 1.96 in pixi.toml.

Commands and flags

  • export qti --attempts is accepted and ignored; attempts come from the assessment record.
  • export typst --form takes one value or all, not a repeated flag.
  • export typst --variant student-report|cohort-report is accepted by the parser; its behavior was not verified, so use report for those documents.
  • init --with-examples writes only an empty bank, banks/example.yaml, and not an example assessment.
  • init writes 8 schema files, not the 4 that docs/guide/setup.md shows.
  • schema prints a modeline path (../.coursebank/schema) that differs from where it writes the files (schema/).
  • --apply exists only on calibrate, and it writes analysis/calibration.yaml, not a bank. The src/main.rs documentation says every command that modifies a bank prints what it would change and requires --apply; the migrate commands use --dry-run instead.
  • There is no import subcommand, although src/main.rs and a bank comment mention a legacy importer.
  • The guide pages use analyze items --assessment exam-2 and similar forms. On the command line the id is positional, and calibrate has no assessment argument. The IRT model value is two-pl, not 2pl.
  • lint --rules, as shown in two guide pages, is lint --list-rules. The lint rule count is 30, not the 26 in one doc comment.
  • migrate counters and migrate ids print follow-up hints that name seal verify and analyze items --all, neither of which exists. Use seal --check to verify.

Analysis

  • IRT in calibrate uses a single administration, so the pooled IRT parameters describe that one administration.
  • dif_flagged is defined, and no code sets it. high_rapid_guess cannot fire because neither reader records response time. mean_response_time_seconds and rapid_guess_rate are never written to the calibration file.
  • Student names are read from the export and do not appear in the stored schema, so a report built from stored data falls back to the student key.

Markup and output

  • #sym.minus, used in an authoring example, is not in the symbol table.
  • Fenced code blocks in stems are not handled by the markup converter.
  • Item figures (assets) go into the Typst payload only, and the bundled Typst templates do not print them. The QTI, practice, and site exports do not emit assets.
  • The bundled Typst templates import @preview packages, which the Typst compiler resolves; whether they are cached or downloaded at compile time was not checked.

Guides and older text

  • The guide pages describe an item-level calibration: block written by calibrate, a version: field on items and placements, letter option ids, and learning_objectives on items. In the 2.0 code the statistics live in analysis/calibration.yaml, a calibration: block in a bank is a validation error, version: is ignored, items use learning_targets, and option ids are names with letters still accepted.
  • docs/guide/setup.md says cognitive_process has eleven values (the code has 19) and tells authors to set order on objectives (which is deprecated and derived from teaches).
  • A docs/AUTHORING.md that a code comment points to does not exist; the file is docs/guide/authoring.md.

Release files

  • No release tags exist, only the rolling nightly tag. The package version is 0.0.0 in Cargo.toml, pixi.toml, and .bumpversion.toml, and CHANGELOG.md has only an [Unreleased] section.
  • RELEASING.md names workflows ci.yml, nightly.yml, release.yml, and docs.yml. The repository has .gitea/workflows/pipeline.yml and sync-readme.yml.
  • pipeline.yml, scripts/check-tag-version.sh, and scripts/dist-attach.sh call scripts/version.sh, which does not exist.
  • The .bumpversion.toml parse pattern requires the first number to start with 1 to 9, so 0.0.0 does not appear to match [CHECK: not run; confirm that pixi run bump works from 0.0.0].
  • The continuous integration and release files were read only shallowly for this README.

Versioning and releases

Releases use calendar versioning in YY.MM.PATCH form. The first release of August 2026 is 26.8.0; a second that month is 26.8.1; the first in September is 26.9.0. Neither the year nor the month is zero-padded, because the version also has to parse as Semantic Versioning (SemVer) and SemVer forbids leading zeros. The number carries no compatibility promise, so treat any release as one that can change behavior and read CHANGELOG.md before upgrading a live course.

pixi run bump cuts the next version and tags it; pushing that tag publishes a release, and every push to main refreshes a rolling nightly prerelease. pixi run bump-show previews the next version without changing anything. The workflows live in .gitea/workflows/; RELEASING.md explains the release flow and what to configure on the Gitea side. The pipeline workflow runs pixi run check first, on pushes to main, tag pushes, pull requests, a daily schedule, and manual dispatch. The nightly job sets COURSEBANK_VERSION from scripts/nightly-version.sh (the last release tag plus the commit count and short hash, or 0.0.0 plus those when there is no tag), builds the binary, packages it as coursebank-nightly-linux-x64.tar.gz, and replaces the nightly prerelease. The release job checks that the tag matches the crate version and publishes a Gitea release with the packaged binary and its checksum. The documentation (rustdoc) is also published for the release and nightly channels.

The release flow has not been used yet, and parts of it are unfinished; Known limits and unfinished parts lists each one under Release files. A build from source or the nightly archive is the way to get a binary until the first tag [CHECK: the sheets confirm that a nightly tag exists but not what archive it holds or whether the job has run successfully].

Documentation

  • docs/guide/setup.md: building a course directory, the references and readings YAML, editor schemas, validate versus lint, exit codes, and reading a course from Rust.
  • docs/guide/authoring.md: bank new, the bank header, the smallest valid item, status gating, the markup, designed distractors, partial credit, the design block, and lint.
  • docs/guide/first_exam.md: one exam end to end, from assemble to calibrate, including after-the-fact fixes.
  • docs/guide/assignment.md: publishing a homework to a Quarto site with export site, password handling, and rotation.
  • docs/guide/recipes.md: short answers on assembly flags, reuse, exports, ingest, analysis, reports, and housekeeping.
  • docs/TYPST.md: the Typst export, markers and slots, typst.yaml, reveal, the lookup order, and the guarantees.
  • These six files are compiled into the API documentation as coursebank::guide modules. They have known drift from the code, listed under Known limits and unfinished parts, so trust this README and the code where they differ.
  • CHANGELOG.md, RELEASING.md, LICENSE.md, and THIRD-PARTY-LICENSES.txt.

The source lives at https://git.scient.ing/education/coursebank. This README is mirrored to a GitHub repository with a notice that the project's home is the Gitea address.

License

coursebank is source-available under the Prosperity Public License 3.0.0. It is not an OSI-approved open-source license. Noncommercial use is free, and the license treats educational institutions, public research organizations, government, and similar noncommercial bodies as noncommercial regardless of how they are funded. Commercial use gets a thirty-day trial; past that it requires a commercial license.

For a commercial license, contact licensing@scient.ing.

Bundled dependencies keep their own licenses, reproduced in THIRD-PARTY-LICENSES.txt in each release.

Contributing

Open an issue first if you want to contribute. Contributed code will need a contributor license agreement so it can ship under both the free and the commercial license. That agreement is not in place yet, so outside patches cannot be merged for now.

S
Description
Author, assemble, administer, and analyze leveled course assessments from YAML item banks
https://coursebank.scient.ing/
Readme
4.9 MiB
0 Stars 1 Watchers 0 Forks
Languages
Rust 97%
Typst 1.8%
Shell 1%