288 lines
10 KiB
Markdown
288 lines
10 KiB
Markdown
# One exam, end to end
|
||
|
||
This follows a single exam from blueprint to student report, using the sequence analysis and docking banks.
|
||
It assumes a course directory with approved items in it; if you do not have one, [`setup`](crate::guide::setup) and [`authoring`](crate::guide::authoring) build one.
|
||
|
||
The loop:
|
||
|
||
```text
|
||
author items ──▶ validate ──▶ lint ──▶ assemble ──▶ export ───┐
|
||
▲ │
|
||
│ administer
|
||
│ │
|
||
calibrate ◀── analyze ◀── ingest ◀───────────────────────────┘
|
||
│
|
||
└──▶ report (students and cohort)
|
||
```
|
||
|
||
The arrow back from `calibrate` to authoring is the point of the design.
|
||
Statistics land on the item, so they are there the next time you consider using it.
|
||
|
||
## Draw a form
|
||
|
||
Describe the exam you want by level, not by item:
|
||
|
||
```console
|
||
$ coursebank assemble exam-2 \
|
||
--title "Exam 2 — Sequence analysis and docking" \
|
||
--kind exam --date 2026-03-24 --platform paper \
|
||
--levels 1=2,2=1,3=2,4=1 --bonus 5=1 \
|
||
--require lo-align-scoring=2,lo-dock-scoring=1 \
|
||
--max-per-bank 4 --cooldown 180 --forms 2 --seed 20260324
|
||
```
|
||
|
||
`--levels 1=2,2=1,3=2,4=1` asks for six scored items across four cognitive levels.
|
||
`--bonus 5=1` adds one level-5 item outside the scored total, which is where level-5 work belongs on a timed multiple-choice paper.
|
||
|
||
`--require` sets floors per objective, so an exam cannot accidentally measure the scoring objective with a single question.
|
||
`--cooldown 180` avoids items used in the last six months, computed by scanning assessment records rather than by consulting a separate ledger.
|
||
There is no ledger file, because a ledger duplicates what the records must already get right and then drifts from it.
|
||
|
||
Use `--dry-run` first.
|
||
It prints the draw without writing anything, and a blueprint that cannot be satisfied tells you which constraint failed rather than silently returning fewer items.
|
||
|
||
What lands in `assessments/exam-2.yaml` is a record of what happened, not a plan:
|
||
|
||
```yaml
|
||
items:
|
||
- number: 1
|
||
item: sequence-analysis::q-align-recall-001
|
||
version: 1
|
||
points: 1.5
|
||
key: [B]
|
||
level: 1
|
||
learning_objectives: [lo-align-algorithm]
|
||
```
|
||
|
||
Level and objectives are denormalized onto the placement so the record reads standalone in five years, whatever the bank says by then.
|
||
|
||
## Check it against the blueprint
|
||
|
||
```console
|
||
$ coursebank assessment show exam-2
|
||
Exam 2 — Sequence analysis and docking 2026-03-24 paper 50 min
|
||
6 scored items, 9.0 points; 1 bonus item, 1.5 points
|
||
levels: 1×2 2×1 3×2 4×1
|
||
estimated time: 41 minutes of 50 allowed
|
||
blueprint: satisfied
|
||
```
|
||
|
||
The time estimate sums each item's `expected_time_seconds`, falling back to a level-based guess for items with no `design` block.
|
||
An exam that does not fit the period is the most common way a well-written exam goes wrong, and it is invisible until you are standing in the room.
|
||
|
||
## Export
|
||
|
||
Two forms with shuffled options, plus a key and a bubble sheet for each:
|
||
|
||
```console
|
||
$ coursebank export typst exam-2 --form all
|
||
wrote build/exam-2-A.typ (from built-in)
|
||
wrote build/exam-2-A-key.typ (from built-in)
|
||
wrote build/exam-2-A-answer-sheet.typ (from built-in)
|
||
wrote build/exam-2-B.typ (from built-in)
|
||
...
|
||
```
|
||
|
||
Option order comes from each form's recorded seed, never from anything stored, so form B's key is generated from the same permutation that produced form B's paper.
|
||
A key that disagrees with its paper is discovered by twenty-five students at once.
|
||
|
||
The `(from built-in)` note means no template override was found.
|
||
`coursebank template dump` writes the defaults into `templates/` so you can restyle them; see [`typst_export`](crate::guide::typst_export).
|
||
|
||
Compile with `pixi run -e docs typst compile build/exam-2-A.typ`.
|
||
|
||
For a Canvas quiz instead:
|
||
|
||
```console
|
||
$ coursebank export qti exam-2 --form A
|
||
wrote build/exam-2-A.zip
|
||
Import in Canvas: Settings -> Import Course Content -> QTI .zip file
|
||
```
|
||
|
||
## Ingest the grading export
|
||
|
||
After the exam, read the grader's output into the response store:
|
||
|
||
```console
|
||
$ coursebank ingest gradescope grading/exam-2/ \
|
||
--assessment exam-2 --form A --pseudonymize --salt-file ~/.coursebank-salt
|
||
read 24 students × 7 items = 168 rows
|
||
wrote data/exam-2-2026s.parquet
|
||
```
|
||
|
||
`--pseudonymize` replaces student identifiers with HMAC pseudonyms keyed by a salt you keep outside the repository.
|
||
Without the salt, hashed ids can be reversed by brute force over a class roster; with the salt committed next to them, so can they.
|
||
The generated `.gitignore` excludes `*.salt` for that reason.
|
||
|
||
Use `--dry-run` on a new export format.
|
||
Gradescope's per-question CSVs vary, and parsing 168 rows wrong is easier to see in a report than in a Parquet file.
|
||
|
||
## Analyze
|
||
|
||
```console
|
||
$ coursebank analyze items --assessment exam-2
|
||
# p rpb flags
|
||
1 0.88 0.21
|
||
2 0.71 0.34
|
||
3 0.46 0.09 low-discrimination
|
||
4 0.63 0.41
|
||
5 0.54 0.18 ambiguous
|
||
6 0.29 -0.12 negative-discrimination
|
||
7 0.21 0.15 bonus
|
||
|
||
reliability: KR-20 = 0.61 (24 examinees, 6 scored items)
|
||
Caution: with 6 items, reliability is limited by test length as much as by
|
||
item quality.
|
||
|
||
3 item(s) need revision
|
||
```
|
||
|
||
Read the corrected point-biserial first.
|
||
It correlates each item against the total of the *other* items, which answers the question you actually care about: did the students who knew the material get this right? A negative value almost always means the key is wrong, so check that before rewriting anything.
|
||
|
||
Item 6 above is the one to look at tonight.
|
||
Item 3's low discrimination is expected if it is an anchor item and worth investigating if it sits at level 3 or higher.
|
||
|
||
Every statistic computed from a class of twenty-four is reported with the caveat it deserves rather than three decimal places of false precision.
|
||
|
||
For a fuller picture:
|
||
|
||
```console
|
||
$ coursebank analyze irt --assessment exam-2 --model 2pl
|
||
$ coursebank analyze students --assessment exam-2
|
||
```
|
||
|
||
## Report
|
||
|
||
```console
|
||
$ coursebank report students --assessment exam-2
|
||
wrote 24 report(s) to reports/exam-2/
|
||
$ coursebank report cohort --assessment exam-2
|
||
wrote reports/exam-2-cohort.md
|
||
```
|
||
|
||
These are two documents with different content, not different tones.
|
||
The student report answers "what should I do next?" and deliberately omits correct answers, other students' data, and any numeric rank.
|
||
Where a student chose a designed distractor, it names the misconception that distractor was built to catch and points at the lecture and slides.
|
||
The cohort report answers "what should I fix?" and holds the item statistics.
|
||
|
||
## Write the statistics back
|
||
|
||
```console
|
||
$ coursebank calibrate --assessment exam-2
|
||
q-align-recall-001 p 0.71 -> 0.68 rpb 0.29 -> 0.31 n 23 -> 47
|
||
q-dock-analyze-002 NEW p 0.29 rpb -0.12 n 24 flag: negative-discrimination
|
||
...
|
||
7 item(s) would change. Re-run with --apply to write.
|
||
```
|
||
|
||
Every command that modifies a bank prints what it would change and requires `--apply`.
|
||
These are reviewed artifacts in a git repository, and a silent rewrite is not something you want to discover in a diff later.
|
||
|
||
```console
|
||
$ coursebank calibrate --assessment exam-2 --apply
|
||
```
|
||
|
||
Now the pooled statistics are on the items, and next term's `assemble` sees them.
|
||
|
||
## When grading reveals a problem
|
||
|
||
Two fields get added to the assessment record by hand, after the fact, and both stay there so that next term's analysis knows the exam was scored the way it was actually scored.
|
||
|
||
An option that turned out to be defensible earns partial credit:
|
||
|
||
```yaml
|
||
- number: 5
|
||
item: structure-and-expression::q-rnaseq-explain-004
|
||
points: 1.5
|
||
key: [B]
|
||
# Decided during grading: option C ("nothing can be said without replicates")
|
||
# is a defensible reading of a descriptive question, so it earns half credit.
|
||
credit_overrides:
|
||
C: 0.5
|
||
```
|
||
|
||
Recording it here rather than editing scores by hand means item analysis sees the same numbers the students did.
|
||
|
||
An item that was broken gets dropped:
|
||
|
||
```yaml
|
||
- number: 6
|
||
item: structure-and-expression::q-dock-analyze-002
|
||
dropped: true
|
||
```
|
||
|
||
Dropped items leave the scored matrix and are not printed on a re-export, but the placement stays in the record, because the fact that the question was asked is part of what happened.
|
||
|
||
Both of these make `analyze items` flag the item as ambiguous, which is the correct outcome.
|
||
The fix is to rewrite the stem so the narrower question is unambiguous, not to relitigate the partial credit every term.
|
||
|
||
## Doing this from Rust
|
||
|
||
The CLI is a thin wrapper.
|
||
Assembling a form programmatically:
|
||
|
||
```rust,no_run
|
||
use std::collections::BTreeMap;
|
||
use std::path::Path;
|
||
|
||
use coursebank::assessment::{Blueprint, History};
|
||
use coursebank::date::Date;
|
||
use coursebank::{select, Catalog, Level};
|
||
|
||
# fn main() -> coursebank::Result<()> {
|
||
let catalog = Catalog::load(Path::new("."))?;
|
||
|
||
let mut level_counts = BTreeMap::new();
|
||
level_counts.insert(Level::Remember, 2);
|
||
level_counts.insert(Level::Understand, 1);
|
||
level_counts.insert(Level::Apply, 2);
|
||
|
||
let blueprint = Blueprint {
|
||
level_counts,
|
||
max_per_bank: Some(4),
|
||
cooldown_days: Some(180),
|
||
seed: Some(20260324),
|
||
..Blueprint::default()
|
||
};
|
||
|
||
// Usage history is derived by scanning the assessment records, so cooldowns are
|
||
// measured against what was actually given rather than a separate ledger.
|
||
let history = History::load(&catalog.layout.assessments())?;
|
||
let selection = select::select(&catalog, &blueprint, &history, Date::new(2026, 3, 24)?)?;
|
||
|
||
for uid in &selection.scored {
|
||
println!("scored: {uid}");
|
||
}
|
||
for note in &selection.notes {
|
||
// Quotas filled by relaxing a constraint say so here.
|
||
println!("note: {note}");
|
||
}
|
||
# Ok(())
|
||
# }
|
||
```
|
||
|
||
Reading responses back and running item analysis:
|
||
|
||
```rust,no_run
|
||
use coursebank::classical::{self, Thresholds};
|
||
use coursebank::store::Store;
|
||
|
||
# fn main() -> coursebank::Result<()> {
|
||
let store = Store::open("data")?;
|
||
// `read` takes an administration id; `read_assessment` gathers every
|
||
// administration of one assessment across terms.
|
||
let responses = store.read_assessment("exam-2")?;
|
||
|
||
let analysis = classical::analyze(&responses, &Thresholds::default(), None, None);
|
||
|
||
for item in analysis.revise_queue() {
|
||
println!(
|
||
"item {}: p = {:.2}, {:?}",
|
||
item.number, item.p_value, item.flags
|
||
);
|
||
}
|
||
# Ok(())
|
||
# }
|
||
```
|