222 lines
8.4 KiB
Markdown
222 lines
8.4 KiB
Markdown
# Authoring items
|
|
|
|
An item is a question plus two things a question does not normally carry: what you predicted about it before anyone answered, and what happened when they did.
|
|
Keeping those next to each other is what turns a pile of questions into an instrument you can improve, because every administration produces a prediction you can check.
|
|
|
|
## A bank
|
|
|
|
```console
|
|
$ coursebank bank new sequence-analysis --title "Sequence analysis"
|
|
wrote banks/sequence-analysis.yaml
|
|
```
|
|
|
|
A bank is a topic grouping, not a unit of reuse. Items are drawn across banks by blueprint, so split banks by whatever makes them easy to edit.
|
|
One per unit is a reasonable default.
|
|
|
|
The header declares scope and defaults:
|
|
|
|
```yaml
|
|
bank:
|
|
id: sequence-analysis
|
|
title: Sequence analysis
|
|
scope:
|
|
units: [u1]
|
|
lectures: [l09, l10, l11]
|
|
|
|
defaults:
|
|
author: Alex Maldonado
|
|
options_per_item: 5
|
|
topics: [alignment]
|
|
```
|
|
|
|
`defaults` fills in fields you would otherwise repeat on every item.
|
|
|
|
## The smallest item that validates
|
|
|
|
```yaml
|
|
items:
|
|
- id: q-align-recall-001
|
|
version: 1
|
|
status: approved
|
|
level: 1
|
|
cognitive_process: recall
|
|
format: single_best_answer
|
|
title: Needleman-Wunsch vs Smith-Waterman
|
|
stem: |
|
|
Which alignment algorithm guarantees an optimal *local* alignment between two sequences?
|
|
options:
|
|
- id: A
|
|
text: Needleman#sym.minus Wunsch
|
|
correct: false
|
|
- id: B
|
|
text: Smith#sym.minus Waterman
|
|
correct: true
|
|
learning_objectives: [lo-align-algorithm]
|
|
sources:
|
|
- lecture: l09
|
|
slides: [12, 13]
|
|
```
|
|
|
|
Ids are never reused and never renumbered.
|
|
The id is the join key that ties an item to every assessment it has appeared on and every response row ever recorded for it, so `q-align-recall-001` stays that even after the stem is rewritten twice.
|
|
|
|
`status` gates assembly.
|
|
Only `approved` items can be drawn onto a graded assessment, and approval requires the item to be fully specified: a cognitive process, an objective, a source, and a key.
|
|
Draft items are visible to `lint` and invisible to `assemble`.
|
|
|
|
`level` and `cognitive_process` are checked against each other.
|
|
`level: 1` with `cognitive_process: evaluate` is an error, not a warning, because one of the two is wrong and the tool cannot tell which.
|
|
|
|
## Markup
|
|
|
|
Stems are written in a small markup that is a subset of Typst with a few Markdown conveniences, because chemistry and biology need subscripts, arrows, and Greek letters, and typing HTML entities into YAML by hand is miserable.
|
|
|
|
```yaml
|
|
stem: >
|
|
A reaction proceeds at 37#sym.degree C with #sym.delta G = #sym.minus 12
|
|
kJ/mol. Rate increases *linearly* with `[S]` below K_m.
|
|
```
|
|
|
|
`#sym.arrow.r`, `#sym.alpha`, `#sym.gt.eq`, and the rest of the table render as arrows and Greek in all three outputs.
|
|
Emphasis uses `*bold*` and `_italic_`, and backticks give monospace.
|
|
The same source becomes HTML for Canvas, Typst for print, and plain text for CSV, so you write it once.
|
|
|
|
## Distractors that earn their place
|
|
|
|
The optional fields on an option are what separate a designed distractor from filler:
|
|
|
|
```yaml
|
|
- id: A
|
|
text: Needleman#sym.minus Wunsch
|
|
correct: false
|
|
misconception: |
|
|
They remember that both are dynamic programming and pick the more familiar name without distinguishing global from local
|
|
error_type: recall_confusion
|
|
explanation: |
|
|
Needleman#sym.minus Wunsch is the global algorithm; it aligns the full length of both sequences.
|
|
feedback_student: |
|
|
Needleman#sym.minus Wunsch is the global algorithm.
|
|
Both use dynamic programming, so the distinction to hold onto is what happens at the matrix boundaries and where the traceback starts.
|
|
```
|
|
|
|
When a third of the cohort picks that option, you know what they were thinking, and the student report can tell each of them specifically rather than saying "incorrect, the answer was B."
|
|
`error_type` is one of thirteen categories, which is what lets cohort analysis say the class is losing points to dropped steps rather than to terminology.
|
|
|
|
`explanation` is for you. `feedback_student` is released to students afterwards and is the text a report shows someone who chose that option. `misconception` is used for both when neither of the others is written, so a partly-authored item degrades gracefully instead of producing a blank.
|
|
|
|
### Partial credit
|
|
|
|
A wrong option that is defensible can earn credit, but only with the argument written down:
|
|
|
|
```yaml
|
|
- id: C
|
|
text: Nothing can be said without replicates
|
|
correct: false
|
|
credit: 0.5
|
|
defensible: true
|
|
defense: >
|
|
A descriptive question about a single pair of libraries admits this
|
|
reading, so it earns half credit rather than zero.
|
|
```
|
|
|
|
`defense` is required whenever credit goes to a wrong option.
|
|
That is deliberate.
|
|
Partial credit decided in the moment and never recorded becomes a decision you cannot reconstruct next term, and then you relitigate it with the next student who asks.
|
|
|
|
The course policy's `partial_credit_floor_level` applies here.
|
|
Credit awarded below that level is flagged, on the theory that a reasonable wrong answer to a recall question means the question is unclear.
|
|
|
|
## Predictions
|
|
|
|
The `design` block is what you think before anyone sits the exam:
|
|
|
|
```yaml
|
|
design:
|
|
expected_difficulty: 0.72
|
|
expected_discrimination: moderate
|
|
expected_time_seconds: 55
|
|
rationale: |
|
|
Recall of a named distinction taught in one slide.
|
|
Most of the cohort should get it; the ones who miss it are confusing the two algorithms rather than failing to recall either.
|
|
```
|
|
|
|
`expected_time_seconds` summed over a form is how you check that an exam fits the period, which is the most common way a well-written exam goes wrong.
|
|
|
|
The other two are checkable predictions.
|
|
After the exam, `lint` compares them against what happened and reports the misses.
|
|
An item you expected to be easy that two thirds of the class missed is either mis-taught or mis-written, and either way you want to be told.
|
|
|
|
## Statistics come back
|
|
|
|
You do not write the `calibration` block.
|
|
`coursebank calibrate` does, after `ingest` and `analyze`:
|
|
|
|
```yaml
|
|
calibration:
|
|
administrations: [exam-2-2026s, exam-2-2025s]
|
|
updated: 2026-04-02
|
|
fingerprint: 8f3a2c...
|
|
n_examinees: 47
|
|
p_value: 0.68
|
|
point_biserial: 0.31
|
|
flags: []
|
|
```
|
|
|
|
Calibration is cumulative rather than per administration.
|
|
Raw per-response data lives in the Parquet tables under `data/`, which are much better at holding it, and the item's YAML keeps the rolled-up estimate plus a list of which administrations went into it.
|
|
Bank files stay readable in a pull request while statistics accumulate across terms.
|
|
Twenty-four students tells you very little; ninety-six across four terms tells you something.
|
|
|
|
The `fingerprint` is why this is safe.
|
|
It covers only what a student saw: the stem, the option text, and the key.
|
|
Retag an item's metadata and the pooled statistics stay valid.
|
|
Reword the stem and the fingerprint changes, the numbers are marked stale, and the linter says so rather than letting you trust a p-value from a question that no longer exists.
|
|
|
|
## Lint before you commit
|
|
|
|
```console
|
|
$ coursebank lint
|
|
banks/sequence-analysis.yaml
|
|
q-align-gap-002 cue-uneven-length the key is 1.8x the average distractor length (94 vs 52)
|
|
q-dock-analyze-002 clarity-stem-length stem runs 84 words
|
|
|
|
2 finding(s)
|
|
```
|
|
|
|
`coursebank lint --rules` lists every rule with its code.
|
|
Silence one you disagree with; the codes exist so that disagreeing is a configuration change rather than a reason to stop running the linter.
|
|
|
|
## Checking a bank from Rust
|
|
|
|
```rust,no_run
|
|
use coursebank::bank::BankFile;
|
|
use coursebank::Status;
|
|
|
|
# fn main() -> coursebank::Result<()> {
|
|
let bank = BankFile::load(std::path::Path::new("banks/sequence-analysis.yaml"))?;
|
|
|
|
let approved = bank
|
|
.items
|
|
.iter()
|
|
.filter(|item| item.status == Status::Approved)
|
|
.count();
|
|
|
|
println!("{approved} of {} items are assemblable", bank.items.len());
|
|
|
|
for item in &bank.items {
|
|
if let Some(calibration) = &item.calibration {
|
|
if !item.calibration_is_current() {
|
|
println!("{}: statistics predate the current wording", item.id);
|
|
} else if let Some(p) = calibration.p_value {
|
|
println!("{}: p = {p:.2}", item.id);
|
|
}
|
|
}
|
|
}
|
|
# Ok(())
|
|
# }
|
|
```
|
|
|
|
## Next
|
|
|
|
[`first_exam`](crate::guide::first_exam) draws a form from this bank and follows it through grading.
|