8.4 KiB
Authoring items
An item is a question plus two things a question does not normally carry: what you predicted about it before anyone answered, and what happened when they did. Keeping those next to each other is what turns a pile of questions into an instrument you can improve, because every administration produces a prediction you can check.
A bank
$ coursebank bank new sequence-analysis --title "Sequence analysis"
wrote banks/sequence-analysis.yaml
A bank is a topic grouping, not a unit of reuse. Items are drawn across banks by blueprint, so split banks by whatever makes them easy to edit. One per unit is a reasonable default.
The header declares scope and defaults:
bank:
id: sequence-analysis
title: Sequence analysis
scope:
units: [u1]
lectures: [l09, l10, l11]
defaults:
author: Alex Maldonado
options_per_item: 5
topics: [alignment]
defaults fills in fields you would otherwise repeat on every item.
The smallest item that validates
items:
- id: q-align-recall-001
version: 1
status: approved
level: 1
cognitive_process: recall
format: single_best_answer
title: Needleman-Wunsch vs Smith-Waterman
stem: |
Which alignment algorithm guarantees an optimal *local* alignment between two sequences?
options:
- id: A
text: Needleman#sym.minus Wunsch
correct: false
- id: B
text: Smith#sym.minus Waterman
correct: true
learning_objectives: [lo-align-algorithm]
sources:
- lecture: l09
slides: [12, 13]
Ids are never reused and never renumbered.
The id is the join key that ties an item to every assessment it has appeared on and every response row ever recorded for it, so q-align-recall-001 stays that even after the stem is rewritten twice.
status gates assembly.
Only approved items can be drawn onto a graded assessment, and approval requires the item to be fully specified: a cognitive process, an objective, a source, and a key.
Draft items are visible to lint and invisible to assemble.
level and cognitive_process are checked against each other.
level: 1 with cognitive_process: evaluate is an error, not a warning, because one of the two is wrong and the tool cannot tell which.
Markup
Stems are written in a small markup that is a subset of Typst with a few Markdown conveniences, because chemistry and biology need subscripts, arrows, and Greek letters, and typing HTML entities into YAML by hand is miserable.
stem: >
A reaction proceeds at 37#sym.degree C with #sym.delta G = #sym.minus 12
kJ/mol. Rate increases *linearly* with `[S]` below K_m.
#sym.arrow.r, #sym.alpha, #sym.gt.eq, and the rest of the table render as arrows and Greek in all three outputs.
Emphasis uses *bold* and _italic_, and backticks give monospace.
The same source becomes HTML for Canvas, Typst for print, and plain text for CSV, so you write it once.
Distractors that earn their place
The optional fields on an option are what separate a designed distractor from filler:
- id: A
text: Needleman#sym.minus Wunsch
correct: false
misconception: |
They remember that both are dynamic programming and pick the more familiar name without distinguishing global from local
error_type: recall_confusion
explanation: |
Needleman#sym.minus Wunsch is the global algorithm; it aligns the full length of both sequences.
feedback_student: |
Needleman#sym.minus Wunsch is the global algorithm.
Both use dynamic programming, so the distinction to hold onto is what happens at the matrix boundaries and where the traceback starts.
When a third of the cohort picks that option, you know what they were thinking, and the student report can tell each of them specifically rather than saying "incorrect, the answer was B."
error_type is one of thirteen categories, which is what lets cohort analysis say the class is losing points to dropped steps rather than to terminology.
explanation is for you. feedback_student is released to students afterwards and is the text a report shows someone who chose that option. misconception is used for both when neither of the others is written, so a partly-authored item degrades gracefully instead of producing a blank.
Partial credit
A wrong option that is defensible can earn credit, but only with the argument written down:
- id: C
text: Nothing can be said without replicates
correct: false
credit: 0.5
defensible: true
defense: >
A descriptive question about a single pair of libraries admits this
reading, so it earns half credit rather than zero.
defense is required whenever credit goes to a wrong option.
That is deliberate.
Partial credit decided in the moment and never recorded becomes a decision you cannot reconstruct next term, and then you relitigate it with the next student who asks.
The course policy's partial_credit_floor_level applies here.
Credit awarded below that level is flagged, on the theory that a reasonable wrong answer to a recall question means the question is unclear.
Predictions
The design block is what you think before anyone sits the exam:
design:
expected_difficulty: 0.72
expected_discrimination: moderate
expected_time_seconds: 55
rationale: |
Recall of a named distinction taught in one slide.
Most of the cohort should get it; the ones who miss it are confusing the two algorithms rather than failing to recall either.
expected_time_seconds summed over a form is how you check that an exam fits the period, which is the most common way a well-written exam goes wrong.
The other two are checkable predictions.
After the exam, lint compares them against what happened and reports the misses.
An item you expected to be easy that two thirds of the class missed is either mis-taught or mis-written, and either way you want to be told.
Statistics come back
You do not write the calibration block.
coursebank calibrate does, after ingest and analyze:
calibration:
administrations: [exam-2-2026s, exam-2-2025s]
updated: 2026-04-02
fingerprint: 8f3a2c...
n_examinees: 47
p_value: 0.68
point_biserial: 0.31
flags: []
Calibration is cumulative rather than per administration.
Raw per-response data lives in the Parquet tables under data/, which are much better at holding it, and the item's YAML keeps the rolled-up estimate plus a list of which administrations went into it.
Bank files stay readable in a pull request while statistics accumulate across terms.
Twenty-four students tells you very little; ninety-six across four terms tells you something.
The fingerprint is why this is safe.
It covers only what a student saw: the stem, the option text, and the key.
Retag an item's metadata and the pooled statistics stay valid.
Reword the stem and the fingerprint changes, the numbers are marked stale, and the linter says so rather than letting you trust a p-value from a question that no longer exists.
Lint before you commit
$ coursebank lint
banks/sequence-analysis.yaml
q-align-gap-002 cue-uneven-length the key is 1.8x the average distractor length (94 vs 52)
q-dock-analyze-002 clarity-stem-length stem runs 84 words
2 finding(s)
coursebank lint --rules lists every rule with its code.
Silence one you disagree with; the codes exist so that disagreeing is a configuration change rather than a reason to stop running the linter.
Checking a bank from Rust
use coursebank::bank::BankFile;
use coursebank::Status;
# fn main() -> coursebank::Result<()> {
let bank = BankFile::load(std::path::Path::new("banks/sequence-analysis.yaml"))?;
let approved = bank
.items
.iter()
.filter(|item| item.status == Status::Approved)
.count();
println!("{approved} of {} items are assemblable", bank.items.len());
for item in &bank.items {
if let Some(calibration) = &item.calibration {
if !item.calibration_is_current() {
println!("{}: statistics predate the current wording", item.id);
} else if let Some(p) = calibration.p_value {
println!("{}: p = {p:.2}", item.id);
}
}
}
# Ok(())
# }
Next
first_exam draws a form from this bank and follows it through grading.