@@ -0,0 +1,221 @@
|
||||
# Authoring items
|
||||
|
||||
An item is a question plus two things a question does not normally carry: what you predicted about it before anyone answered, and what happened when they did.
|
||||
Keeping those next to each other is what turns a pile of questions into an instrument you can improve, because every administration produces a prediction you can check.
|
||||
|
||||
## A bank
|
||||
|
||||
```console
|
||||
$ coursebank bank new sequence-analysis --title "Sequence analysis"
|
||||
wrote banks/sequence-analysis.yaml
|
||||
```
|
||||
|
||||
A bank is a topic grouping, not a unit of reuse. Items are drawn across banks by blueprint, so split banks by whatever makes them easy to edit.
|
||||
One per unit is a reasonable default.
|
||||
|
||||
The header declares scope and defaults:
|
||||
|
||||
```yaml
|
||||
bank:
|
||||
id: sequence-analysis
|
||||
title: Sequence analysis
|
||||
scope:
|
||||
units: [u1]
|
||||
lectures: [l09, l10, l11]
|
||||
|
||||
defaults:
|
||||
author: Alex Maldonado
|
||||
options_per_item: 5
|
||||
topics: [alignment]
|
||||
```
|
||||
|
||||
`defaults` fills in fields you would otherwise repeat on every item.
|
||||
|
||||
## The smallest item that validates
|
||||
|
||||
```yaml
|
||||
items:
|
||||
- id: q-align-recall-001
|
||||
version: 1
|
||||
status: approved
|
||||
level: 1
|
||||
cognitive_process: recall
|
||||
format: single_best_answer
|
||||
title: Needleman-Wunsch vs Smith-Waterman
|
||||
stem: |
|
||||
Which alignment algorithm guarantees an optimal *local* alignment between two sequences?
|
||||
options:
|
||||
- id: A
|
||||
text: Needleman#sym.minus Wunsch
|
||||
correct: false
|
||||
- id: B
|
||||
text: Smith#sym.minus Waterman
|
||||
correct: true
|
||||
learning_objectives: [lo-align-algorithm]
|
||||
sources:
|
||||
- lecture: l09
|
||||
slides: [12, 13]
|
||||
```
|
||||
|
||||
Ids are never reused and never renumbered.
|
||||
The id is the join key that ties an item to every assessment it has appeared on and every response row ever recorded for it, so `q-align-recall-001` stays that even after the stem is rewritten twice.
|
||||
|
||||
`status` gates assembly.
|
||||
Only `approved` items can be drawn onto a graded assessment, and approval requires the item to be fully specified: a cognitive process, an objective, a source, and a key.
|
||||
Draft items are visible to `lint` and invisible to `assemble`.
|
||||
|
||||
`level` and `cognitive_process` are checked against each other.
|
||||
`level: 1` with `cognitive_process: evaluate` is an error, not a warning, because one of the two is wrong and the tool cannot tell which.
|
||||
|
||||
## Markup
|
||||
|
||||
Stems are written in a small markup that is a subset of Typst with a few Markdown conveniences, because chemistry and biology need subscripts, arrows, and Greek letters, and typing HTML entities into YAML by hand is miserable.
|
||||
|
||||
```yaml
|
||||
stem: >
|
||||
A reaction proceeds at 37#sym.degree C with #sym.delta G = #sym.minus 12
|
||||
kJ/mol. Rate increases *linearly* with `[S]` below K_m.
|
||||
```
|
||||
|
||||
`#sym.arrow.r`, `#sym.alpha`, `#sym.gt.eq`, and the rest of the table render as arrows and Greek in all three outputs.
|
||||
Emphasis uses `*bold*` and `_italic_`, and backticks give monospace.
|
||||
The same source becomes HTML for Canvas, Typst for print, and plain text for CSV, so you write it once.
|
||||
|
||||
## Distractors that earn their place
|
||||
|
||||
The optional fields on an option are what separate a designed distractor from filler:
|
||||
|
||||
```yaml
|
||||
- id: A
|
||||
text: Needleman#sym.minus Wunsch
|
||||
correct: false
|
||||
misconception: |
|
||||
They remember that both are dynamic programming and pick the more familiar name without distinguishing global from local
|
||||
error_type: recall_confusion
|
||||
explanation: |
|
||||
Needleman#sym.minus Wunsch is the global algorithm; it aligns the full length of both sequences.
|
||||
feedback_student: |
|
||||
Needleman#sym.minus Wunsch is the global algorithm.
|
||||
Both use dynamic programming, so the distinction to hold onto is what happens at the matrix boundaries and where the traceback starts.
|
||||
```
|
||||
|
||||
When a third of the cohort picks that option, you know what they were thinking, and the student report can tell each of them specifically rather than saying "incorrect, the answer was B."
|
||||
`error_type` is one of thirteen categories, which is what lets cohort analysis say the class is losing points to dropped steps rather than to terminology.
|
||||
|
||||
`explanation` is for you. `feedback_student` is released to students afterwards and is the text a report shows someone who chose that option. `misconception` is used for both when neither of the others is written, so a partly-authored item degrades gracefully instead of producing a blank.
|
||||
|
||||
### Partial credit
|
||||
|
||||
A wrong option that is defensible can earn credit, but only with the argument written down:
|
||||
|
||||
```yaml
|
||||
- id: C
|
||||
text: Nothing can be said without replicates
|
||||
correct: false
|
||||
credit: 0.5
|
||||
defensible: true
|
||||
defense: >
|
||||
A descriptive question about a single pair of libraries admits this
|
||||
reading, so it earns half credit rather than zero.
|
||||
```
|
||||
|
||||
`defense` is required whenever credit goes to a wrong option.
|
||||
That is deliberate.
|
||||
Partial credit decided in the moment and never recorded becomes a decision you cannot reconstruct next term, and then you relitigate it with the next student who asks.
|
||||
|
||||
The course policy's `partial_credit_floor_level` applies here.
|
||||
Credit awarded below that level is flagged, on the theory that a reasonable wrong answer to a recall question means the question is unclear.
|
||||
|
||||
## Predictions
|
||||
|
||||
The `design` block is what you think before anyone sits the exam:
|
||||
|
||||
```yaml
|
||||
design:
|
||||
expected_difficulty: 0.72
|
||||
expected_discrimination: moderate
|
||||
expected_time_seconds: 55
|
||||
rationale: |
|
||||
Recall of a named distinction taught in one slide.
|
||||
Most of the cohort should get it; the ones who miss it are confusing the two algorithms rather than failing to recall either.
|
||||
```
|
||||
|
||||
`expected_time_seconds` summed over a form is how you check that an exam fits the period, which is the most common way a well-written exam goes wrong.
|
||||
|
||||
The other two are checkable predictions.
|
||||
After the exam, `lint` compares them against what happened and reports the misses.
|
||||
An item you expected to be easy that two thirds of the class missed is either mis-taught or mis-written, and either way you want to be told.
|
||||
|
||||
## Statistics come back
|
||||
|
||||
You do not write the `calibration` block.
|
||||
`coursebank calibrate` does, after `ingest` and `analyze`:
|
||||
|
||||
```yaml
|
||||
calibration:
|
||||
administrations: [exam-2-2026s, exam-2-2025s]
|
||||
updated: 2026-04-02
|
||||
fingerprint: 8f3a2c...
|
||||
n_examinees: 47
|
||||
p_value: 0.68
|
||||
point_biserial: 0.31
|
||||
flags: []
|
||||
```
|
||||
|
||||
Calibration is cumulative rather than per administration.
|
||||
Raw per-response data lives in the Parquet tables under `data/`, which are much better at holding it, and the item's YAML keeps the rolled-up estimate plus a list of which administrations went into it.
|
||||
Bank files stay readable in a pull request while statistics accumulate across terms.
|
||||
Twenty-four students tells you very little; ninety-six across four terms tells you something.
|
||||
|
||||
The `fingerprint` is why this is safe.
|
||||
It covers only what a student saw: the stem, the option text, and the key.
|
||||
Retag an item's metadata and the pooled statistics stay valid.
|
||||
Reword the stem and the fingerprint changes, the numbers are marked stale, and the linter says so rather than letting you trust a p-value from a question that no longer exists.
|
||||
|
||||
## Lint before you commit
|
||||
|
||||
```console
|
||||
$ coursebank lint
|
||||
banks/sequence-analysis.yaml
|
||||
q-align-gap-002 cue-uneven-length the key is 1.8x the average distractor length (94 vs 52)
|
||||
q-dock-analyze-002 clarity-stem-length stem runs 84 words
|
||||
|
||||
2 finding(s)
|
||||
```
|
||||
|
||||
`coursebank lint --rules` lists every rule with its code.
|
||||
Silence one you disagree with; the codes exist so that disagreeing is a configuration change rather than a reason to stop running the linter.
|
||||
|
||||
## Checking a bank from Rust
|
||||
|
||||
```rust,no_run
|
||||
use coursebank::bank::BankFile;
|
||||
use coursebank::Status;
|
||||
|
||||
# fn main() -> coursebank::Result<()> {
|
||||
let bank = BankFile::load(std::path::Path::new("banks/sequence-analysis.yaml"))?;
|
||||
|
||||
let approved = bank
|
||||
.items
|
||||
.iter()
|
||||
.filter(|item| item.status == Status::Approved)
|
||||
.count();
|
||||
|
||||
println!("{approved} of {} items are assemblable", bank.items.len());
|
||||
|
||||
for item in &bank.items {
|
||||
if let Some(calibration) = &item.calibration {
|
||||
if !item.calibration_is_current() {
|
||||
println!("{}: statistics predate the current wording", item.id);
|
||||
} else if let Some(p) = calibration.p_value {
|
||||
println!("{}: p = {p:.2}", item.id);
|
||||
}
|
||||
}
|
||||
}
|
||||
# Ok(())
|
||||
# }
|
||||
```
|
||||
|
||||
## Next
|
||||
|
||||
[`first_exam`](crate::guide::first_exam) draws a form from this bank and follows it through grading.
|
||||
@@ -0,0 +1,288 @@
|
||||
# One exam, end to end
|
||||
|
||||
This follows a single exam from blueprint to student report, using the sequence analysis and docking banks.
|
||||
It assumes a course directory with approved items in it; if you do not have one, [`setup`](crate::guide::setup) and [`authoring`](crate::guide::authoring) build one.
|
||||
|
||||
The loop:
|
||||
|
||||
```text
|
||||
author items ──▶ validate ──▶ lint ──▶ assemble ──▶ export ───┐
|
||||
▲ │
|
||||
│ administer
|
||||
│ │
|
||||
calibrate ◀── analyze ◀── ingest ◀───────────────────────────┘
|
||||
│
|
||||
└──▶ report (students and cohort)
|
||||
```
|
||||
|
||||
The arrow back from `calibrate` to authoring is the point of the design.
|
||||
Statistics land on the item, so they are there the next time you consider using it.
|
||||
|
||||
## Draw a form
|
||||
|
||||
Describe the exam you want by level, not by item:
|
||||
|
||||
```console
|
||||
$ coursebank assemble exam-2 \
|
||||
--title "Exam 2 — Sequence analysis and docking" \
|
||||
--kind exam --date 2026-03-24 --platform paper \
|
||||
--levels 1=2,2=1,3=2,4=1 --bonus 5=1 \
|
||||
--require lo-align-scoring=2,lo-dock-scoring=1 \
|
||||
--max-per-bank 4 --cooldown 180 --forms 2 --seed 20260324
|
||||
```
|
||||
|
||||
`--levels 1=2,2=1,3=2,4=1` asks for six scored items across four cognitive levels.
|
||||
`--bonus 5=1` adds one level-5 item outside the scored total, which is where level-5 work belongs on a timed multiple-choice paper.
|
||||
|
||||
`--require` sets floors per objective, so an exam cannot accidentally measure the scoring objective with a single question.
|
||||
`--cooldown 180` avoids items used in the last six months, computed by scanning assessment records rather than by consulting a separate ledger.
|
||||
There is no ledger file, because a ledger duplicates what the records must already get right and then drifts from it.
|
||||
|
||||
Use `--dry-run` first.
|
||||
It prints the draw without writing anything, and a blueprint that cannot be satisfied tells you which constraint failed rather than silently returning fewer items.
|
||||
|
||||
What lands in `assessments/exam-2.yaml` is a record of what happened, not a plan:
|
||||
|
||||
```yaml
|
||||
items:
|
||||
- number: 1
|
||||
item: sequence-analysis::q-align-recall-001
|
||||
version: 1
|
||||
points: 1.5
|
||||
key: [B]
|
||||
level: 1
|
||||
learning_objectives: [lo-align-algorithm]
|
||||
```
|
||||
|
||||
Level and objectives are denormalized onto the placement so the record reads standalone in five years, whatever the bank says by then.
|
||||
|
||||
## Check it against the blueprint
|
||||
|
||||
```console
|
||||
$ coursebank assessment show exam-2
|
||||
Exam 2 — Sequence analysis and docking 2026-03-24 paper 50 min
|
||||
6 scored items, 9.0 points; 1 bonus item, 1.5 points
|
||||
levels: 1×2 2×1 3×2 4×1
|
||||
estimated time: 41 minutes of 50 allowed
|
||||
blueprint: satisfied
|
||||
```
|
||||
|
||||
The time estimate sums each item's `expected_time_seconds`, falling back to a level-based guess for items with no `design` block.
|
||||
An exam that does not fit the period is the most common way a well-written exam goes wrong, and it is invisible until you are standing in the room.
|
||||
|
||||
## Export
|
||||
|
||||
Two forms with shuffled options, plus a key and a bubble sheet for each:
|
||||
|
||||
```console
|
||||
$ coursebank export typst exam-2 --form all
|
||||
wrote build/exam-2-A.typ (from built-in)
|
||||
wrote build/exam-2-A-key.typ (from built-in)
|
||||
wrote build/exam-2-A-answer-sheet.typ (from built-in)
|
||||
wrote build/exam-2-B.typ (from built-in)
|
||||
...
|
||||
```
|
||||
|
||||
Option order comes from each form's recorded seed, never from anything stored, so form B's key is generated from the same permutation that produced form B's paper.
|
||||
A key that disagrees with its paper is discovered by twenty-five students at once.
|
||||
|
||||
The `(from built-in)` note means no template override was found.
|
||||
`coursebank template dump` writes the defaults into `templates/` so you can restyle them; see [`typst_export`](crate::guide::typst_export).
|
||||
|
||||
Compile with `pixi run -e docs typst compile build/exam-2-A.typ`.
|
||||
|
||||
For a Canvas quiz instead:
|
||||
|
||||
```console
|
||||
$ coursebank export qti exam-2 --form A
|
||||
wrote build/exam-2-A.zip
|
||||
Import in Canvas: Settings -> Import Course Content -> QTI .zip file
|
||||
```
|
||||
|
||||
## Ingest the grading export
|
||||
|
||||
After the exam, read the grader's output into the response store:
|
||||
|
||||
```console
|
||||
$ coursebank ingest gradescope grading/exam-2/ \
|
||||
--assessment exam-2 --form A --pseudonymize --salt-file ~/.coursebank-salt
|
||||
read 24 students × 7 items = 168 rows
|
||||
wrote data/exam-2-2026s.parquet
|
||||
```
|
||||
|
||||
`--pseudonymize` replaces student identifiers with HMAC pseudonyms keyed by a salt you keep outside the repository.
|
||||
Without the salt, hashed ids can be reversed by brute force over a class roster; with the salt committed next to them, so can they.
|
||||
The generated `.gitignore` excludes `*.salt` for that reason.
|
||||
|
||||
Use `--dry-run` on a new export format.
|
||||
Gradescope's per-question CSVs vary, and parsing 168 rows wrong is easier to see in a report than in a Parquet file.
|
||||
|
||||
## Analyze
|
||||
|
||||
```console
|
||||
$ coursebank analyze items --assessment exam-2
|
||||
# p rpb flags
|
||||
1 0.88 0.21
|
||||
2 0.71 0.34
|
||||
3 0.46 0.09 low-discrimination
|
||||
4 0.63 0.41
|
||||
5 0.54 0.18 ambiguous
|
||||
6 0.29 -0.12 negative-discrimination
|
||||
7 0.21 0.15 bonus
|
||||
|
||||
reliability: KR-20 = 0.61 (24 examinees, 6 scored items)
|
||||
Caution: with 6 items, reliability is limited by test length as much as by
|
||||
item quality.
|
||||
|
||||
3 item(s) need revision
|
||||
```
|
||||
|
||||
Read the corrected point-biserial first.
|
||||
It correlates each item against the total of the *other* items, which answers the question you actually care about: did the students who knew the material get this right? A negative value almost always means the key is wrong, so check that before rewriting anything.
|
||||
|
||||
Item 6 above is the one to look at tonight.
|
||||
Item 3's low discrimination is expected if it is an anchor item and worth investigating if it sits at level 3 or higher.
|
||||
|
||||
Every statistic computed from a class of twenty-four is reported with the caveat it deserves rather than three decimal places of false precision.
|
||||
|
||||
For a fuller picture:
|
||||
|
||||
```console
|
||||
$ coursebank analyze irt --assessment exam-2 --model 2pl
|
||||
$ coursebank analyze students --assessment exam-2
|
||||
```
|
||||
|
||||
## Report
|
||||
|
||||
```console
|
||||
$ coursebank report students --assessment exam-2
|
||||
wrote 24 report(s) to reports/exam-2/
|
||||
$ coursebank report cohort --assessment exam-2
|
||||
wrote reports/exam-2-cohort.md
|
||||
```
|
||||
|
||||
These are two documents with different content, not different tones.
|
||||
The student report answers "what should I do next?" and deliberately omits correct answers, other students' data, and any numeric rank.
|
||||
Where a student chose a designed distractor, it names the misconception that distractor was built to catch and points at the lecture and slides.
|
||||
The cohort report answers "what should I fix?" and holds the item statistics.
|
||||
|
||||
## Write the statistics back
|
||||
|
||||
```console
|
||||
$ coursebank calibrate --assessment exam-2
|
||||
q-align-recall-001 p 0.71 -> 0.68 rpb 0.29 -> 0.31 n 23 -> 47
|
||||
q-dock-analyze-002 NEW p 0.29 rpb -0.12 n 24 flag: negative-discrimination
|
||||
...
|
||||
7 item(s) would change. Re-run with --apply to write.
|
||||
```
|
||||
|
||||
Every command that modifies a bank prints what it would change and requires `--apply`.
|
||||
These are reviewed artifacts in a git repository, and a silent rewrite is not something you want to discover in a diff later.
|
||||
|
||||
```console
|
||||
$ coursebank calibrate --assessment exam-2 --apply
|
||||
```
|
||||
|
||||
Now the pooled statistics are on the items, and next term's `assemble` sees them.
|
||||
|
||||
## When grading reveals a problem
|
||||
|
||||
Two fields get added to the assessment record by hand, after the fact, and both stay there so that next term's analysis knows the exam was scored the way it was actually scored.
|
||||
|
||||
An option that turned out to be defensible earns partial credit:
|
||||
|
||||
```yaml
|
||||
- number: 5
|
||||
item: structure-and-expression::q-rnaseq-explain-004
|
||||
points: 1.5
|
||||
key: [B]
|
||||
# Decided during grading: option C ("nothing can be said without replicates")
|
||||
# is a defensible reading of a descriptive question, so it earns half credit.
|
||||
credit_overrides:
|
||||
C: 0.5
|
||||
```
|
||||
|
||||
Recording it here rather than editing scores by hand means item analysis sees the same numbers the students did.
|
||||
|
||||
An item that was broken gets dropped:
|
||||
|
||||
```yaml
|
||||
- number: 6
|
||||
item: structure-and-expression::q-dock-analyze-002
|
||||
dropped: true
|
||||
```
|
||||
|
||||
Dropped items leave the scored matrix and are not printed on a re-export, but the placement stays in the record, because the fact that the question was asked is part of what happened.
|
||||
|
||||
Both of these make `analyze items` flag the item as ambiguous, which is the correct outcome.
|
||||
The fix is to rewrite the stem so the narrower question is unambiguous, not to relitigate the partial credit every term.
|
||||
|
||||
## Doing this from Rust
|
||||
|
||||
The CLI is a thin wrapper.
|
||||
Assembling a form programmatically:
|
||||
|
||||
```rust,no_run
|
||||
use std::collections::BTreeMap;
|
||||
use std::path::Path;
|
||||
|
||||
use coursebank::assessment::Blueprint;
|
||||
use coursebank::history::History;
|
||||
use coursebank::date::Date;
|
||||
use coursebank::{select, Catalog, Level};
|
||||
|
||||
# fn main() -> coursebank::Result<()> {
|
||||
let catalog = Catalog::load(Path::new("."))?;
|
||||
|
||||
let mut level_counts = BTreeMap::new();
|
||||
level_counts.insert(Level::Remember, 2);
|
||||
level_counts.insert(Level::Understand, 1);
|
||||
level_counts.insert(Level::Apply, 2);
|
||||
|
||||
let blueprint = Blueprint {
|
||||
level_counts,
|
||||
max_per_bank: Some(4),
|
||||
cooldown_days: Some(180),
|
||||
seed: Some(20260324),
|
||||
..Blueprint::default()
|
||||
};
|
||||
|
||||
// Usage history is derived by scanning the assessment records, so cooldowns are
|
||||
// measured against what was actually given rather than a separate ledger.
|
||||
let history = History::load(&catalog.layout.assessments())?;
|
||||
let selection = select::select(&catalog, &blueprint, &history, Date::new(2026, 3, 24)?)?;
|
||||
|
||||
for uid in &selection.scored {
|
||||
println!("scored: {uid}");
|
||||
}
|
||||
for note in &selection.notes {
|
||||
// Quotas filled by relaxing a constraint say so here.
|
||||
println!("note: {note}");
|
||||
}
|
||||
# Ok(())
|
||||
# }
|
||||
```
|
||||
|
||||
Reading responses back and running item analysis:
|
||||
|
||||
```rust,no_run
|
||||
use coursebank::classical::{self, Thresholds};
|
||||
use coursebank::store::Store;
|
||||
|
||||
# fn main() -> coursebank::Result<()> {
|
||||
let store = Store::open("data")?;
|
||||
// `read` takes an administration id; `read_assessment` gathers every
|
||||
// administration of one assessment across terms.
|
||||
let responses = store.read_assessment("exam-2")?;
|
||||
|
||||
let analysis = classical::analyze(&responses, &Thresholds::default(), None, None);
|
||||
|
||||
for item in analysis.revise_queue() {
|
||||
println!(
|
||||
"item {}: p = {:.2}, {:?}",
|
||||
item.number, item.p_value, item.flags
|
||||
);
|
||||
}
|
||||
# Ok(())
|
||||
# }
|
||||
```
|
||||
@@ -0,0 +1,119 @@
|
||||
# Recipes
|
||||
|
||||
Short answers, for when you know the shape of the tool and want the invocation.
|
||||
|
||||
## Assembly
|
||||
|
||||
**Draw only from material I have taught.**
|
||||
`--lectures l09,l10,l11`.
|
||||
Combine with `--topics` and `--banks` to narrow further; the filters intersect.
|
||||
|
||||
**See the draw before committing to it.**
|
||||
`--dry-run`.
|
||||
Prints the selection and any notes about constraints that had to bend, and writes nothing.
|
||||
|
||||
**Reproduce a draw exactly.**
|
||||
`--seed N`.
|
||||
The same seed against the same pool gives the same items in the same order.
|
||||
Recorded in the assessment file, so a draw stays reproducible after the fact.
|
||||
|
||||
**Two forms that differ only in option order.**
|
||||
`--forms 2`.
|
||||
Each form gets its own seed; item order is shared unless the form sets `shuffle_items`.
|
||||
|
||||
**A blueprint I cannot satisfy.**
|
||||
The error names the level, how many items were asked for, and how many were available after filtering.
|
||||
Usually the fix is a shorter cooldown or a wider lecture range, not more items.
|
||||
|
||||
## Reuse
|
||||
|
||||
```console
|
||||
$ coursebank usage history q-align-recall-001
|
||||
$ coursebank usage unused
|
||||
```
|
||||
|
||||
`unused` lists approved items never placed on an assessment, which is the queue of work you already did and forgot about.
|
||||
|
||||
## Exports
|
||||
|
||||
**A printable exam.**
|
||||
`coursebank export typst exam-2 --form all`.
|
||||
Add `--variant key` to write only the key.
|
||||
|
||||
**A Canvas quiz.**
|
||||
`coursebank export qti exam-2 --form A`.
|
||||
Add `--no-feedback` to leave per-option feedback out of the package.
|
||||
|
||||
**A Markdown copy for a colleague to read.**
|
||||
`coursebank export md exam-2`.
|
||||
Add `--with-key` for the answers and rationales.
|
||||
|
||||
**Restyle the printed output.**
|
||||
`coursebank template dump`, then edit `templates/exam.typ`.
|
||||
See [`typst_export`](crate::guide::typst_export).
|
||||
|
||||
## Ingest
|
||||
|
||||
**Gradescope.**
|
||||
`coursebank ingest gradescope grading/exam-2/ --assessment exam-2`.
|
||||
Point it at the directory holding the per-question CSVs.
|
||||
|
||||
**Canvas.**
|
||||
`coursebank ingest canvas export.csv --assessment exam-2`.
|
||||
The Student Analysis export, not the gradebook.
|
||||
|
||||
**Keep student identities out of the repository.**
|
||||
`--pseudonymize --salt-file ~/.coursebank-salt`.
|
||||
Keep the salt outside the repository; the point of the salt is that hashed ids cannot be brute-forced over a class roster, which fails if the salt sits next to them.
|
||||
|
||||
**Check a parse before writing.** `--dry-run`.
|
||||
|
||||
## Analysis
|
||||
|
||||
| Question | Command |
|
||||
|:--|:--|
|
||||
| Which items misbehaved? | `analyze items --assessment exam-2` |
|
||||
| How hard is each item, on a common scale? | `analyze irt --assessment exam-2 --model 2pl` |
|
||||
| Which students are struggling, and with what? | `analyze students --assessment exam-2` |
|
||||
| What is in the store? | `data` |
|
||||
|
||||
Pool across terms by passing the assessment id rather than one administration id.
|
||||
Twenty-four students supports very little; ninety-six across four terms supports something.
|
||||
|
||||
## Reports
|
||||
|
||||
```console
|
||||
$ coursebank report students --assessment exam-2
|
||||
$ coursebank report cohort --assessment exam-2
|
||||
```
|
||||
|
||||
The student report omits correct answers, other students' data, and any rank.
|
||||
Hand it out without a second pass.
|
||||
|
||||
## After grading
|
||||
|
||||
**An option turned out to be defensible.**
|
||||
Add `credit_overrides: {C: 0.5}` to the placement in the assessment record.
|
||||
Do not edit scores by hand, or item analysis sees different numbers than the students did.
|
||||
|
||||
**An item was broken.**
|
||||
Add `dropped: true` to the placement.
|
||||
It leaves the scored matrix and is not printed on re-export, but the record of having asked it stays.
|
||||
|
||||
**Statistics onto the items.**
|
||||
`coursebank calibrate --assessment exam-2`, read the diff, then `--apply`.
|
||||
|
||||
## Housekeeping
|
||||
|
||||
**Editor validation stopped working.**
|
||||
`coursebank schema` rewrites the JSON Schemas.
|
||||
They ship with the binary, so an upgrade can leave them stale.
|
||||
|
||||
**A pre-commit hook.**
|
||||
`coursebank validate && coursebank lint`.
|
||||
Exit code `2` means findings, `1` means the command failed, so a hook can treat them differently.
|
||||
|
||||
**Build without Parquet.**
|
||||
`pixi run build-lean`.
|
||||
The response store falls back to CSV.
|
||||
Useful if you want a binary with a shorter dependency list; the tradeoff is slower reads on large stores.
|
||||
@@ -0,0 +1,188 @@
|
||||
# Setting up a course
|
||||
|
||||
A course is a directory in a git repository.
|
||||
Nothing lives in a database, and the tool holds no state of its own, so a course you set up in 2026 opens in 2031 with whatever version of coursebank you have then.
|
||||
|
||||
## Make the directory
|
||||
|
||||
```console
|
||||
$ coursebank init --code "BIOSC 1540" --title "Computational Biology" --term 2026s
|
||||
wrote course.yaml
|
||||
wrote 4 JSON Schema files
|
||||
```
|
||||
|
||||
That gives you:
|
||||
|
||||
| Path | Holds | Written by |
|
||||
|:--|:--|:--|
|
||||
| `course.yaml` | identity, grading policy, objectives, lectures | you |
|
||||
| `banks/` | items, with design intent and pooled statistics | you, then `calibrate` |
|
||||
| `assessments/` | what was given, to whom, when | `assemble`, then you |
|
||||
| `data/` | one row per student per item | `ingest` |
|
||||
| `reports/` | generated Markdown and HTML | `report` |
|
||||
| `build/` | exports: QTI packages, `.typ` files, PDFs | `export` |
|
||||
| `schema/` | JSON Schemas for editor validation | `init`, `schema` |
|
||||
|
||||
`build/` and `reports/` are in the generated `.gitignore`.
|
||||
The other four are the repository's content and belong in review.
|
||||
|
||||
Add `--with-examples` if you want a filled-in bank to read rather than an empty directory to stare at.
|
||||
|
||||
## Point your editor at the schemas
|
||||
|
||||
The schemas are the difference between authoring items and looking up field names.
|
||||
With them wired in, your editor completes `cognitive_process` from the eleven legal values and underlines a typo in `learning_objectives` as you type.
|
||||
|
||||
Put the modeline at the top of each file:
|
||||
|
||||
```yaml
|
||||
# yaml-language-server: $schema=../schema/bank.schema.json
|
||||
```
|
||||
|
||||
`coursebank schema` reprints the paths and the exact line to paste.
|
||||
Rerun it after upgrading, since the schemas ship with the binary.
|
||||
|
||||
## Fill in `course.yaml`
|
||||
|
||||
Four registries live here, and everything else references them by id.
|
||||
|
||||
### Policy
|
||||
|
||||
Conventions stated once instead of per assessment:
|
||||
|
||||
```yaml
|
||||
policy:
|
||||
points_per_item: 1.5
|
||||
options_per_item: 5
|
||||
bonus_levels: [5]
|
||||
allow_partial_credit: true
|
||||
partial_credit_floor_level: 3
|
||||
mastery_threshold: 0.75
|
||||
min_items_for_mastery: 3
|
||||
```
|
||||
|
||||
`partial_credit_floor_level: 3` is the one to think about.
|
||||
Below Apply, a defensible wrong answer usually means the item is unclear rather than that the student partly understood something.
|
||||
Setting a floor makes that a rule you decided once, so it stops being an argument you have every term with a student at your desk.
|
||||
|
||||
### Units and lectures
|
||||
|
||||
Units are the coarse grouping.
|
||||
Lectures carry a date and belong to a unit:
|
||||
|
||||
```yaml
|
||||
units:
|
||||
- id: u1
|
||||
title: Sequence analysis
|
||||
description: Alignment, scoring models, and database search.
|
||||
|
||||
lectures:
|
||||
l09:
|
||||
title: Pairwise alignment
|
||||
date: 2026-02-10
|
||||
unit: u1
|
||||
readings:
|
||||
- "Durbin et al., ch. 2"
|
||||
```
|
||||
|
||||
The dates are what let a student report say which lecture to review, and what lets `assemble --lectures l09,l10` draw only from material you have taught.
|
||||
|
||||
### Learning objectives
|
||||
|
||||
The load-bearing registry.
|
||||
An objective's wording lives in exactly one place, so rewording it updates every report that quotes it:
|
||||
|
||||
```yaml
|
||||
learning_objectives:
|
||||
lo-align-algorithm:
|
||||
text: Trace the dynamic programming recurrence for a global or local alignment and explain what each term contributes.
|
||||
unit: u1
|
||||
lectures: [l09]
|
||||
level_ceiling: 4
|
||||
tags: [algorithms]
|
||||
|
||||
lo-align-scoring:
|
||||
text: Predict how changing a substitution matrix or gap penalty changes the resulting alignment.
|
||||
unit: u1
|
||||
lectures: [l10]
|
||||
prerequisites: [lo-align-algorithm]
|
||||
level_ceiling: 4
|
||||
```
|
||||
|
||||
Write the text in the second person and start with a verb, because reports quote it verbatim to students.
|
||||
|
||||
Three fields do work later that is easy to miss now.
|
||||
`prerequisites` is walked backwards by student reports to suggest where to start reviewing, so a student who missed the scoring objective gets pointed at the algorithm first.
|
||||
`level_ceiling` is the highest level you intend to assess the objective at; placing an item above it is a warning, which means either the item overreaches or the ceiling needs raising, and both are useful to be asked about.
|
||||
`assessed: false` marks an objective you teach but measure some other way, such as by project rubric, which stops coverage reporting from flagging it as a gap on every run.
|
||||
|
||||
Objective ids are join keys.
|
||||
Renaming one orphans every item and every stored response that referenced it, so pick names you can live with.
|
||||
|
||||
### Stimuli
|
||||
|
||||
A shared passage, table, or figure that several items ask about:
|
||||
|
||||
````yaml
|
||||
stimuli:
|
||||
s-dock-poses:
|
||||
body: |
|
||||
A docking run produces five poses of the same ligand. Scores are in
|
||||
kcal/mol; RMSD is measured against the crystallographic pose.
|
||||
|
||||
```
|
||||
Pose Score RMSD (Å) Cluster size
|
||||
1 -9.8 6.2 3
|
||||
2 -9.4 1.1 28
|
||||
```
|
||||
caption: Docking output for a single ligand against one receptor.
|
||||
````
|
||||
|
||||
Items reference it with `stimulus: s-dock-poses`.
|
||||
Declaring it here rather than pasting it into four items means a correction to the table happens once.
|
||||
|
||||
## Check it
|
||||
|
||||
```console
|
||||
$ coursebank validate
|
||||
course.yaml: ok
|
||||
banks: 0 files, 0 items
|
||||
assessments: 0 records
|
||||
```
|
||||
|
||||
`validate` enforces what must be true: every reference resolves, ids are unique, keys are present, credit is in range.
|
||||
It reports everything wrong in one pass instead of one problem per run, because fixing one typo per invocation is not a workflow.
|
||||
|
||||
`lint` is separate and advises on what is usually a mistake: an option that gives away the answer by being longer than the others, a stem with no task in it, a level that disagrees with the cognitive process.
|
||||
Every rule has a code you can silence.
|
||||
The split matters because a linter that blocks a commit for a style opinion gets disabled, and then you lose the validator with it.
|
||||
|
||||
Exit codes are meaningful.
|
||||
`0` means success, `1` means the command failed, and `2` means validation or linting found something.
|
||||
|
||||
## Where to go next
|
||||
|
||||
[`authoring`](crate::guide::authoring) writes the first bank.
|
||||
[`first_exam`](crate::guide::first_exam) takes an exam from blueprint to student report.
|
||||
|
||||
## Reading a course from Rust
|
||||
|
||||
The CLI is one caller.
|
||||
[`Catalog::load`](crate::catalog::Catalog::load) reads a whole course directory and indexes every item by global id:
|
||||
|
||||
```rust,no_run
|
||||
use std::path::Path;
|
||||
|
||||
use coursebank::Catalog;
|
||||
|
||||
# fn main() -> coursebank::Result<()> {
|
||||
let catalog = Catalog::load(Path::new("path/to/course"))?;
|
||||
|
||||
println!("{} items across {} banks", catalog.entries.len(), catalog.banks.len());
|
||||
|
||||
// Global ids are `bank::item`. Bare ids resolve when unambiguous.
|
||||
let entry = catalog.require("sequence-analysis::q-align-recall-001")?;
|
||||
println!("{}", entry.item.display_title());
|
||||
# Ok(())
|
||||
# }
|
||||
```
|
||||
Reference in New Issue
Block a user