coursebank/docs/guide/first_exam.md

One exam, end to end

This follows a single exam from blueprint to student report, using the sequence analysis and docking banks. It assumes a course directory with approved items in it; if you do not have one, setup and authoring build one.

The loop:

author items ──▶ validate ──▶ lint ──▶ assemble ──▶ export ───┐
     ▲                                                        │
     │                                                   administer
     │                                                        │
 calibrate ◀── analyze ◀── ingest ◀───────────────────────────┘
     │
     └──▶ report (students and cohort)

The arrow back from calibrate to authoring is the point of the design. Statistics land on the item, so they are there the next time you consider using it.

Draw a form

Describe the exam you want by level, not by item:

$ coursebank assemble exam-2 \
    --title "Exam 2 — Sequence analysis and docking" \
    --kind exam --date 2026-03-24 --platform paper \
    --levels 1=2,2=1,3=2,4=1 --bonus 5=1 \
    --require lo-align-scoring=2,lo-dock-scoring=1 \
    --max-per-bank 4 --cooldown 180 --forms 2 --seed 20260324

--levels 1=2,2=1,3=2,4=1 asks for six scored items across four cognitive levels. --bonus 5=1 adds one level-5 item outside the scored total, which is where level-5 work belongs on a timed multiple-choice paper.

--require sets floors per objective, so an exam cannot accidentally measure the scoring objective with a single question. --cooldown 180 avoids items used in the last six months, computed by scanning assessment records rather than by consulting a separate ledger. There is no ledger file, because a ledger duplicates what the records must already get right and then drifts from it.

Use --dry-run first. It prints the draw without writing anything, and a blueprint that cannot be satisfied tells you which constraint failed rather than silently returning fewer items.

What lands in assessments/exam-2.yaml is a record of what happened, not a plan:

items:
  - number: 1
    item: sequence-analysis::q-align-recall-001
    version: 1
    points: 1.5
    key: [B]
    level: 1
    learning_objectives: [lo-align-algorithm]

Level and objectives are denormalized onto the placement so the record reads standalone in five years, whatever the bank says by then.

Check it against the blueprint

$ coursebank assessment show exam-2
Exam 2 — Sequence analysis and docking   2026-03-24   paper   50 min
  6 scored items, 9.0 points; 1 bonus item, 1.5 points
  levels: 1×2  2×1  3×2  4×1
  estimated time: 41 minutes of 50 allowed
  blueprint: satisfied

The time estimate sums each item's expected_time_seconds, falling back to a level-based guess for items with no design block. An exam that does not fit the period is the most common way a well-written exam goes wrong, and it is invisible until you are standing in the room.

Export

Two forms with shuffled options, plus a key and a bubble sheet for each:

$ coursebank export typst exam-2 --form all
wrote build/exam-2-A.typ (from built-in)
wrote build/exam-2-A-key.typ (from built-in)
wrote build/exam-2-A-answer-sheet.typ (from built-in)
wrote build/exam-2-B.typ (from built-in)
...

Option order comes from each form's recorded seed, never from anything stored, so form B's key is generated from the same permutation that produced form B's paper. A key that disagrees with its paper is discovered by twenty-five students at once.

The (from built-in) note means no template override was found. coursebank template dump writes the defaults into templates/ so you can restyle them; see typst_export.

Compile with pixi run -e docs typst compile build/exam-2-A.typ.

For a Canvas quiz instead:

$ coursebank export qti exam-2 --form A
wrote build/exam-2-A.zip
Import in Canvas: Settings -> Import Course Content -> QTI .zip file

Ingest the grading export

After the exam, read the grader's output into the response store:

$ coursebank ingest gradescope grading/exam-2/ \
    --assessment exam-2 --form A --pseudonymize --salt-file ~/.coursebank-salt
read 24 students × 7 items = 168 rows
wrote data/exam-2-2026s.parquet

--pseudonymize replaces student identifiers with HMAC pseudonyms keyed by a salt you keep outside the repository. Without the salt, hashed ids can be reversed by brute force over a class roster; with the salt committed next to them, so can they. The generated .gitignore excludes *.salt for that reason.

Use --dry-run on a new export format. Gradescope's per-question CSVs vary, and parsing 168 rows wrong is easier to see in a report than in a Parquet file.

Analyze

$ coursebank analyze items --assessment exam-2
#  p      rpb    flags
1  0.88   0.21
2  0.71   0.34
3  0.46   0.09   low-discrimination
4  0.63   0.41
5  0.54   0.18   ambiguous
6  0.29  -0.12   negative-discrimination
7  0.21   0.15   bonus

reliability: KR-20 = 0.61 (24 examinees, 6 scored items)
  Caution: with 6 items, reliability is limited by test length as much as by
  item quality.

3 item(s) need revision

Read the corrected point-biserial first. It correlates each item against the total of the other items, which answers the question you actually care about: did the students who knew the material get this right? A negative value almost always means the key is wrong, so check that before rewriting anything.

Item 6 above is the one to look at tonight. Item 3's low discrimination is expected if it is an anchor item and worth investigating if it sits at level 3 or higher.

Every statistic computed from a class of twenty-four is reported with the caveat it deserves rather than three decimal places of false precision.

For a fuller picture:

$ coursebank analyze irt --assessment exam-2 --model 2pl
$ coursebank analyze students --assessment exam-2

Report

$ coursebank report students --assessment exam-2
wrote 24 report(s) to reports/exam-2/
$ coursebank report cohort --assessment exam-2
wrote reports/exam-2-cohort.md

These are two documents with different content, not different tones. The student report answers "what should I do next?" and deliberately omits correct answers, other students' data, and any numeric rank. Where a student chose a designed distractor, it names the misconception that distractor was built to catch and points at the lecture and slides. The cohort report answers "what should I fix?" and holds the item statistics.

Write the statistics back

$ coursebank calibrate --assessment exam-2
q-align-recall-001   p 0.71 -> 0.68   rpb 0.29 -> 0.31   n 23 -> 47
q-dock-analyze-002   NEW    p 0.29    rpb -0.12         n 24   flag: negative-discrimination
...
7 item(s) would change. Re-run with --apply to write.

Every command that modifies a bank prints what it would change and requires --apply. These are reviewed artifacts in a git repository, and a silent rewrite is not something you want to discover in a diff later.

$ coursebank calibrate --assessment exam-2 --apply

Now the pooled statistics are on the items, and next term's assemble sees them.

When grading reveals a problem

Two fields get added to the assessment record by hand, after the fact, and both stay there so that next term's analysis knows the exam was scored the way it was actually scored.

An option that turned out to be defensible earns partial credit:

  - number: 5
    item: structure-and-expression::q-rnaseq-explain-004
    points: 1.5
    key: [B]
    # Decided during grading: option C ("nothing can be said without replicates")
    # is a defensible reading of a descriptive question, so it earns half credit.
    credit_overrides:
      C: 0.5

Recording it here rather than editing scores by hand means item analysis sees the same numbers the students did.

An item that was broken gets dropped:

  - number: 6
    item: structure-and-expression::q-dock-analyze-002
    dropped: true

Dropped items leave the scored matrix and are not printed on a re-export, but the placement stays in the record, because the fact that the question was asked is part of what happened.

Both of these make analyze items flag the item as ambiguous, which is the correct outcome. The fix is to rewrite the stem so the narrower question is unambiguous, not to relitigate the partial credit every term.

Doing this from Rust

The CLI is a thin wrapper. Assembling a form programmatically:

use std::collections::BTreeMap;
use std::path::Path;

use coursebank::assessment::Blueprint;
use coursebank::history::History;
use coursebank::date::Date;
use coursebank::{select, Catalog, Level};

# fn main() -> coursebank::Result<()> {
let catalog = Catalog::load(Path::new("."))?;

let mut level_counts = BTreeMap::new();
level_counts.insert(Level::Remember, 2);
level_counts.insert(Level::Understand, 1);
level_counts.insert(Level::Apply, 2);

let blueprint = Blueprint {
    level_counts,
    max_per_bank: Some(4),
    cooldown_days: Some(180),
    seed: Some(20260324),
    ..Blueprint::default()
};

// Usage history is derived by scanning the assessment records, so cooldowns are
// measured against what was actually given rather than a separate ledger.
let history = History::load(&catalog.layout.assessments())?;
let selection = select::select(&catalog, &blueprint, &history, Date::new(2026, 3, 24)?)?;

for uid in &selection.scored {
    println!("scored: {uid}");
}
for note in &selection.notes {
    // Quotas filled by relaxing a constraint say so here.
    println!("note: {note}");
}
# Ok(())
# }

Reading responses back and running item analysis:

use coursebank::classical::{self, Thresholds};
use coursebank::store::Store;

# fn main() -> coursebank::Result<()> {
let store = Store::open("data")?;
// `read` takes an administration id; `read_assessment` gathers every
// administration of one assessment across terms.
let responses = store.read_assessment("exam-2")?;

let analysis = classical::analyze(&responses, &Thresholds::default(), None, None);

for item in analysis.revise_queue() {
    println!(
        "item {}: p = {:.2}, {:?}",
        item.number, item.p_value, item.flags
    );
}
# Ok(())
# }