Replace the Prosperity Public License 3.0.0 with the Apache License 2.0: LICENSE.md, a new NOTICE.md, the Cargo license field, every SPDX header, and the README's license and contributing sections.
1153 lines
95 KiB
Markdown
1153 lines
95 KiB
Markdown
# coursebank
|
|
|
|
coursebank runs the assessment side of a course as version-controlled data.
|
|
Questions, exams, grading exports, and the statistics computed from them live as YAML (a plain-text data format) and Parquet (a binary table format) files in a git repository, so they can be reviewed in pull requests and tracked as each question is reused across terms.
|
|
|
|
It is both a command-line tool and a Rust library.
|
|
The command line covers the whole loop: author items, validate and lint them, assemble an exam, export it for print or Canvas, ingest the grading export, analyze it, and write the results back so the next assembly knows how each question has behaved.
|
|
The write-back step is narrower than it sounds: `calibrate --apply` writes the pooled statistics to `analysis/calibration.yaml`, and it never edits a bank file.
|
|
A `calibration:` block inside a bank is a validation error.
|
|
|
|
This README describes what the code does.
|
|
The code wins where an older guide, a doc comment, or the `--help` text says something different, and [Known limits and unfinished parts](#known-limits-and-unfinished-parts) lists the differences.
|
|
|
|
## Contents
|
|
|
|
- [The problem and the design commitments](#the-problem-and-the-design-commitments)
|
|
- [The files](#the-files)
|
|
- [Installing and building](#installing-and-building)
|
|
- [Quick start](#quick-start)
|
|
- [The loop, end to end](#the-loop-end-to-end)
|
|
- [Authoring items](#authoring-items)
|
|
- [Assembling an exam](#assembling-an-exam)
|
|
- [Reuse history](#reuse-history)
|
|
- [Sealing before printing](#sealing-before-printing)
|
|
- [Exporting](#exporting)
|
|
- [Ingesting grading data](#ingesting-grading-data)
|
|
- [Analysis and calibration](#analysis-and-calibration)
|
|
- [Reports](#reports)
|
|
- [Lectures, references, and site content](#lectures-references-and-site-content)
|
|
- [Migrating an older course](#migrating-an-older-course)
|
|
- [Command reference](#command-reference)
|
|
- [Conventions that apply to every command](#conventions-that-apply-to-every-command)
|
|
- [Using coursebank as a Rust library](#using-coursebank-as-a-rust-library)
|
|
- [Worked picture: biosc1000](#worked-picture-biosc1000)
|
|
- [Worked picture: biosc1540](#worked-picture-biosc1540)
|
|
- [What the two repositories teach about adopting the tool](#what-the-two-repositories-teach-about-adopting-the-tool)
|
|
- [Known limits and unfinished parts](#known-limits-and-unfinished-parts)
|
|
- [Versioning and releases](#versioning-and-releases)
|
|
- [Documentation](#documentation)
|
|
- [License](#license)
|
|
- [Contributing](#contributing)
|
|
|
|
## The problem and the design commitments
|
|
|
|
An exam is usually a document, and the information that makes it worth reusing is scattered: which questions discriminated between strong and weak students, which wrong answer attracted students who held a particular misconception, which question was used last term.
|
|
The crate documentation in `src/lib.rs` states the premise directly: questions, exams, and grading exports are worth treating as a dataset rather than as a pile of documents.
|
|
Treating them that way makes four routine tasks cheap: knowing which questions discriminate, catching a poorly worded item from its distractor pattern, telling a student which misconception their wrong answer indicates, and not reusing a question three terms in a row.
|
|
|
|
The design rests on a few commitments.
|
|
|
|
- Records are the single source of truth for reuse: no separate ledger file exists. The reuse history of a question is derived by scanning every assessment record for its id. The records hold the answer to when a question was last used, and the answer cannot disagree with them.
|
|
- A fingerprint covers only what a student saw: the fingerprint of an item is computed from its stem, its options (with which are correct), and its stimulus. Tags, topics, and other metadata do not contribute. Retagging an item keeps its pooled statistics valid, while rewording the stem changes the fingerprint and marks the statistics stale.
|
|
- Validation reports everything at once: `validate` collects every problem and prints them together, so you fix a file in one pass instead of one error at a time.
|
|
- Validation and linting are separate: `validate` finds problems that must be fixed. `lint` finds weaknesses in item craft, and every lint rule has a code that you can silence.
|
|
- Small samples are labeled: an analysis of fewer than 100 examinees says so, and states the standard error it uses.
|
|
- No database and no hidden state: a course is a directory in a git repository. The tool holds no state of its own, so a course set up in 2026 opens in 2031 with whatever version of coursebank you have then.
|
|
|
|
## The files
|
|
|
|
A course is a directory of four kinds of file:
|
|
|
|
| Path | Holds |
|
|
|------|-------|
|
|
| `course.yaml` | course identity, policy, learning objectives, lectures |
|
|
| `banks/*.yaml` | items, with their design intent |
|
|
| `assessments/*.yaml` | what was given, in what order, on what date |
|
|
| `data/*.parquet` | one row per student per item |
|
|
|
|
The first three are hand-editable YAML meant to be read in a diff.
|
|
The response data is machine-written and stored in Parquet, so pandas, polars, DuckDB, and R can read it without this tool.
|
|
Writing comma-separated values (CSV) instead is available with `ingest --format csv`, and reading accepts either format regardless of which one wrote the file.
|
|
|
|
The pooled statistics for each item (difficulty, discrimination, per-option selection rates, flags, and any item response theory (IRT) parameters) live in a fifth file, `analysis/calibration.yaml`, written only by `calibrate --apply`.
|
|
A command that loads the course attaches those statistics to each item by id, so the statistics are available wherever the item is used, though the bank file never contains them.
|
|
|
|
The files `course.yaml`, `references.yaml`, `glossary.yaml`, `lectures/*.yaml`, and `objectives/*.yaml` are one course file split by subject, and a course that keeps everything in `course.yaml` loads unchanged.
|
|
The full directory layout is:
|
|
|
|
| Path | Role | Written by | Ignored by `init`'s `.gitignore` |
|
|
|------|------|-----------|---------|
|
|
| `course.yaml` | identity, policy, units | you | no |
|
|
| `references.yaml` | works cited | you | no |
|
|
| `glossary.yaml` | terms, equations, values, and misconceptions no single objective owns | you, or `migrate glossary` | no |
|
|
| `lectures/*.yaml` | one lecture each: readings, materials, and what it teaches | you | no |
|
|
| `objectives/*.yaml` | one learning objective each, with its learning targets and the glossary entries they first need | you | no |
|
|
| `banks/*.yaml` | item banks | you | no |
|
|
| `assessments/*.yaml` | assessment records | `assemble`, then you | no |
|
|
| `seals/*.yaml` | frozen copy of what was printed | `seal` | no |
|
|
| `analysis/calibration.yaml` | pooled statistics per item | `calibrate --apply` | no (meant to be committed) |
|
|
| `analysis/administrations/*.csv` | cohort aggregates for one administration | `analyze items --record` | no (meant to be committed) |
|
|
| `data/*.parquet` or `*.csv` | one row per student per item | `ingest` | yes |
|
|
| `reports/` | per-student and cohort reports | `report` | yes |
|
|
| `build/` | generated exports | `export` | yes |
|
|
| `schema/` | JSON (JavaScript Object Notation) Schemas for editor validation | `init`, `schema` | no |
|
|
| `templates/` | Typst templates and `typst.yaml` | `template dump`, `template config` | no |
|
|
|
|
`coursebank course files` lists the files the course is assembled from, `coursebank course build` prints the merged course, and `coursebank course where ID` says which file defines a given unit, lecture, objective, target, reference, stimulus, term, equation, value, or misconception id.
|
|
Every command merges the fragments in memory, and each id may be defined in only one place.
|
|
`course build` is for reading and for other programs: nothing in coursebank reads what it writes.
|
|
|
|
`coursebank init` writes a `.gitignore` that excludes `build/`, `reports/`, `data/`, `*.pdf`, `*.salt`, `.coursebank-salt`, `.DS_Store`, and `*.swp`.
|
|
`init` keeps an existing `.gitignore` and inserts only the missing patterns at the top, under the comment `# Added by coursebank init.`
|
|
The directory split is deliberate: `data/` and `reports/` carry one row or file per student and stay out of git, while `analysis/` holds cohort aggregates only and is meant to be committed.
|
|
|
|
## Installing and building
|
|
|
|
coursebank builds from source.
|
|
The sources do not record any other install route [CHECK: confirm there is no packaged install other than the nightly and release archives described under Versioning and releases].
|
|
The binary needs no runtime setup and reads no environment variables.
|
|
The only environment variable in the crate is `COURSEBANK_VERSION`, read at build time: when it is set and non-blank, it replaces the version that `--version` prints, and otherwise the package version is used.
|
|
|
|
The Rust requirement is stated in two places, and they differ.
|
|
`Cargo.toml` declares `rust-version = "1.85"` and edition 2024, so Rust 1.85 or newer is the minimum the crate asks for.
|
|
`pixi.toml` pins the toolchain to `rust >=1.96.0,<1.97`, so a pixi environment builds with a newer compiler than the minimum.
|
|
|
|
The repository uses [pixi](https://pixi.sh) to pin the toolchain and wrap the common tasks:
|
|
|
|
```sh
|
|
pixi run build # release binary at target/release/coursebank
|
|
pixi run install # install the binary onto your PATH
|
|
pixi run tests
|
|
pixi run check # verify: formatting, lint, tests, and doc build
|
|
```
|
|
|
|
`pixi task list` prints every task with a one-line description.
|
|
`pixi run install` runs `cargo install --path . --locked`.
|
|
`pixi run cb validate -C path/to/course` runs the command-line tool through cargo without installing it.
|
|
`pixi run check` runs the format check, clippy with warnings as errors, the full test suite, the documentation examples, a check that every code fence in `docs/` declares its language, and the rustdoc build.
|
|
|
|
Plain cargo works too with a recent toolchain:
|
|
|
|
```sh
|
|
cargo build --release
|
|
```
|
|
|
|
Two tools that coursebank does not bundle are needed to turn some exports into finished documents.
|
|
coursebank writes Typst source (`.typ`) but does not compile it, so a Typst compiler must be installed separately.
|
|
The pixi `docs` environment adds `typst`, and the hint that `export typst` prints is `pixi run -e docs typst compile <file>.typ`.
|
|
The bundled Typst templates import the packages `@preview/mitex:0.2.7` and, for student reports, `@preview/whalogen:0.3.0` from the Typst package registry [CHECK: whether the compiler downloads these at compile time or finds them cached was not checked].
|
|
Quarto renders the `.qmd` files from `export practice` and `export site`, and the partials from `lecture readings` and `lecture objectives` are written for a Quarto site.
|
|
`export qti` needs nothing beyond the binary.
|
|
|
|
The Cargo manifest defines no feature flags.
|
|
The help text for `export site`, the `pixi run build-lean` task, and several guide pages describe `parquet` and `site` features. No `[features]` table exists in `Cargo.toml`, and no source file gates code on a feature.
|
|
Parquet storage and `export site` are always compiled in, and instructions such as `cargo build --features site` refer to a feature that the manifest does not define.
|
|
|
|
## Quick start
|
|
|
|
Create a course, validate it, then assemble and export an exam from the pool:
|
|
|
|
```sh
|
|
coursebank init --code "BIOSC 1540" --title "Computational Biology" --term 2026s
|
|
coursebank validate
|
|
coursebank assemble exam-4 --levels 1=6,2=8,3=10,4=6 --forms 2
|
|
coursebank export typst exam-4 --form A
|
|
```
|
|
|
|
`init` refuses to overwrite an existing `course.yaml`.
|
|
It writes a skeleton `course.yaml` (one unit, one lecture, one objective, and one target as placeholders to rename), the JSON Schemas in `schema/`, and the `.gitignore` described above.
|
|
`assemble` draws from the pool of approved items, so on a new course with no approved items it fails with a message that no approved items match the blueprint.
|
|
|
|
Run `coursebank --help` for the full command set.
|
|
|
|
## The loop, end to end
|
|
|
|
The loop is: author items, validate and lint them, assemble an exam, export it, seal it before printing, give the exam, ingest the grading export, analyze it, and calibrate so that the statistics are stored for the next assembly.
|
|
The sequence below is built from the commands in the guide pages and in the two course repositories, in the order a real exam follows [CHECK: none of these commands was run for this README; the flags come from the code and the course scripts, so run the sequence on a scratch course before relying on it].
|
|
The assessment id `exam-2` and the objective ids are examples.
|
|
|
|
```sh
|
|
# 1. Author: create an empty bank, then write items in banks/sequence-analysis.yaml
|
|
coursebank bank new sequence-analysis --title "Sequence analysis"
|
|
|
|
# 2. Check: validate must pass, lint is advice
|
|
coursebank validate
|
|
coursebank lint
|
|
coursebank catalog --coverage
|
|
|
|
# 3. Assemble: draw two forms from the approved pool
|
|
coursebank assemble exam-2 --title "Exam 2" --kind exam --date 2026-03-24 \
|
|
--platform paper --levels 1=2,2=1,3=2,4=1 --bonus 5=1 \
|
|
--require lo-align-scoring=2,lo-dock-scoring=1 \
|
|
--max-per-bank 4 --cooldown 180 --forms 2 --seed 20260324
|
|
coursebank assessment show exam-2
|
|
|
|
# 4. Export: Typst source for each form, then compile it with typst
|
|
coursebank export typst exam-2 --form all
|
|
|
|
# 5. Seal: freeze what is about to be printed, before printing
|
|
coursebank seal exam-2
|
|
|
|
# 6. Give the exam and grade it in Gradescope, one export directory per form.
|
|
|
|
# 7. Ingest: names are read but not stored; pseudonymize with a salt kept outside the repo
|
|
coursebank ingest gradescope A=grading/exam-2/form-a B=grading/exam-2/form-b \
|
|
--assessment exam-2 --pseudonymize --salt-file path/outside/the/repo/salt --dry-run
|
|
coursebank ingest gradescope A=grading/exam-2/form-a B=grading/exam-2/form-b \
|
|
--assessment exam-2 --pseudonymize --salt-file path/outside/the/repo/salt
|
|
coursebank data
|
|
|
|
# 8. Analyze: look first, decide any drop or regrade, then record the aggregates
|
|
coursebank analyze items exam-2
|
|
coursebank analyze items exam-2 --record
|
|
coursebank analyze irt exam-2
|
|
coursebank analyze students exam-2
|
|
coursebank report cohort exam-2 --typst
|
|
coursebank report students exam-2 --typst
|
|
|
|
# 9. Calibrate once, after every administration is ingested, then lint again
|
|
coursebank calibrate
|
|
coursebank calibrate --apply
|
|
coursebank lint
|
|
```
|
|
|
|
A few orderings matter.
|
|
|
|
- Seal after export and before printing: the seal records the printed letters and the digests of what was printed. Sealing after ingest records drift as if it were the truth.
|
|
- Decide drops and regrades before `--record`: `analyze items --record` writes two administration files that are never rewritten, and it refuses to overwrite them. Edit the assessment record first (`dropped: true`, `dropped_as: full_credit`, or `credit_overrides`) when a question must be dropped or regraded, and do not edit scores by hand.
|
|
- Calibrate once, after the loop: `calibrate` pools every stored administration, so running it after each exam recomputes the same pool and running it before the last ingest leaves that exam out.
|
|
- Lint again after calibrating: five lint rules compare the observed statistics with the author's predictions in the `design` block, and they have nothing to compare until `calibrate --apply` has written `analysis/calibration.yaml`.
|
|
- Pass the assessment id as an argument: the guide pages show forms such as `analyze items --assessment exam-2`; the command line takes the id as a positional argument, and `calibrate` takes no assessment at all.
|
|
|
|
## Authoring items
|
|
|
|
### What an item is
|
|
|
|
An item is a question plus two things a question does not normally carry: what you predicted about it before anyone answered, and what happened when they did.
|
|
The prediction lives in the item (the `design` block).
|
|
The outcome lives in `analysis/calibration.yaml`, keyed by item id.
|
|
|
|
An item has these fields (unknown keys are errors, so a misspelled field is caught):
|
|
|
|
- `id`, `status`, `level`, `stem` (required), and `format` (default `single_best_answer`).
|
|
- `cognitive_process`, `title`, `stimulus`, `points`, `bonus`, `topics`, `prerequisites`, `author`, `notes_private`.
|
|
- `options`, each with `id`, `text`, `correct`, `credit`, `explanation`, `hint`, `misconception`, `error_type`, `feedback_student`, `selection_rate_expected`, `defensible`, `defense`, and `retired`.
|
|
- `learning_targets` (alias `learning_objectives`), the targets the item is tagged to.
|
|
- `sources`, each naming a `lecture` with optional `slides`, `readings`, and `recording_seconds`.
|
|
- `solution`, with `model_answer`, `explanation`, `rubric`, `accepted`, and `review` citations.
|
|
- `assets`, each with `path`, `alt`, and `caption`.
|
|
- `design`, with `expected_difficulty` (0 to 1), `expected_discrimination` (`low`, `moderate`, or `high`), `expected_time_seconds`, and `rationale`.
|
|
- `review` (`reviewed_by`, `reviewed_on`, `action`, `notes`) and a `retired` block (`on`, `reason`, `replaced_by`).
|
|
- `supersedes`, for an item that replaces another.
|
|
|
|
Three conventions differ from the older guide pages.
|
|
Statistics are never in a bank.
|
|
The `version:` and `history:` fields on an item are ignored and never written, and `migrate stems` removes them.
|
|
Items are tagged with `learning_targets`, and the older name `learning_objectives` is accepted as an alias.
|
|
|
|
Options form a pool.
|
|
An item may hold several defensible keys and more distractors than any form shows, and `assemble` draws which are shown.
|
|
Option ids are names of the form `o-` followed by lowercase letters, digits, and hyphens (for example `o-fourth-line`), and the letters `A` to `H` from earlier versions are still accepted.
|
|
An option id identifies an option; it is not its printed position, because shuffled forms relabel options on output.
|
|
|
|
### Levels, formats, and status
|
|
|
|
Levels follow the revised Bloom taxonomy and are written as integers: 1 Remember, 2 Understand, 3 Apply, 4 Analyze, and 5 Evaluate or Create.
|
|
A `cognitive_process` must belong to the item's level; the code defines 19 values:
|
|
|
|
| Level | Cognitive processes |
|
|
|-------|--------------------|
|
|
| 1 | `recognize`, `recall` |
|
|
| 2 | `interpret`, `exemplify`, `classify`, `summarize`, `infer`, `compare`, `explain` |
|
|
| 3 | `execute`, `implement` |
|
|
| 4 | `differentiate`, `organize`, `attribute` |
|
|
| 5 | `check`, `critique`, `generate`, `plan`, `produce` |
|
|
|
|
The four formats are `single_best_answer`, `multiple_response`, `true_false`, and `open_response`.
|
|
An `open_response` item has no options and is not machine-scored.
|
|
Status is one of `draft`, `in_review`, `needs_revision`, `approved`, and `retired`, and only an `approved` item without a `retired` block is eligible for assembly.
|
|
An approved item must also have a `cognitive_process`, at least one learning target, at least one source, and a `design` block, and an approved `open_response` item needs a `solution` with a `model_answer` or a `rubric`.
|
|
|
|
The `error_type` of a distractor names the kind of mistake it was written to catch.
|
|
The 13 values are `recall_confusion`, `terminology_swap`, `misconception`, `incomplete_understanding`, `overgeneralization`, `plausible_irrelevant`, `dropped_step`, `reversed_relationship`, `wrong_procedure`, `sign_or_magnitude_error`, `ignores_interacting_condition`, `correct_but_incomplete`, and `common_shortcut`.
|
|
The text a student sees for a chosen option comes from `feedback_student` first, then `misconception`, then `explanation`.
|
|
That ordering allows a student report to tell a student which misconception their wrong answer indicates.
|
|
|
|
Partial credit on an option needs a `defense` and `defensible: true`, and it must respect the course policy (`allow_partial_credit`, and `partial_credit_floor_level`).
|
|
|
|
### Writing stems and options
|
|
|
|
Stems and options use a subset of Typst with some Markdown conveniences: `*bold*`, `_italic_`, backticks for inline code, `#sub[x]` and `#sup[x]`, and a fixed table of `#sym.` symbols such as `#sym.arrow.r`, `#sym.times`, and `#sym.degree`.
|
|
Math goes between dollar signs as LaTeX and is handed to the Typst package mitex.
|
|
Chemistry written as `\ce{...}` goes to the whalogen package, which the template must import, and the exporter warns if it does not.
|
|
The markup is converted per target: HTML for QTI (Question and Test Interoperability) and the site, Unicode for Markdown and plain text, and Typst for printed documents.
|
|
The conversion is not identical everywhere.
|
|
A single-asterisk `*x*` is emphasis in HTML and QTI. In Typst output it is bold.
|
|
|
|
### Course-level structure
|
|
|
|
Learning objectives and learning targets form two tiers.
|
|
An objective is what a report classifies, and a target is what items are tagged to.
|
|
An id may exist in only one of the two registries, and an item that tags an objective that has targets is a validation error: tag the target.
|
|
Lectures list the objectives they `teach`, and readings attach to lectures with a `ref`, a `locator`, a `role` (`assigned` or `supplemental`), and the targets they support.
|
|
The `policy` block carries defaults such as `points_per_item` (1.0), `options_per_item` (4), `mastery_threshold` (0.75), `min_items_for_mastery` (2), `allow_partial_credit` (true), `bonus_levels`, and a `grade_scale`.
|
|
Schema 2.1 adds the course's names for the five levels (`levels`), its `exam_conditions`, named `blueprints`, and `target_text_forbid`, patterns that objective and target text must not match; `exam_conditions` and `blueprints` are stored and validated, and no command reads them yet.
|
|
|
|
Schema 2.1 also adds four registries beside the objectives and targets: terms (`term-`), equations (`eq-`), values (`val-`), and misconceptions (`mis-`).
|
|
Each entry is defined once, in the objective file whose targets first need it or in `glossary.yaml`, and a target lists the entries it turns on and carries a student-facing `explanation`.
|
|
Student-facing prose names an entry in braces, such as `{term-gap-penalty}`, and every page renders the reference.
|
|
A bank option names the misconception it encodes with `misconception_id`.
|
|
The guide's `glossary` page has the fields, the brace table, and the example.
|
|
|
|
### Editor support
|
|
|
|
`init` and `schema` write nine JSON Schemas into `schema/`: `course`, `references`, `glossary`, `lecture`, `objective`, `calibration`, `measurements`, `bank`, and `assessment`.
|
|
Put a modeline at the top of a YAML file and an editor with a YAML language server validates the file as you type:
|
|
|
|
```yaml
|
|
# yaml-language-server: $schema=../schema/bank.schema.json
|
|
```
|
|
|
|
The schemas are written by hand for autocompletion, and the crate's own validation remains the authority.
|
|
The `schema` command prints a modeline that points at `../.coursebank/schema`, which is not where the schemas are written.
|
|
Use the `../schema/` form shown above.
|
|
|
|
### What validate checks, and what lint checks
|
|
|
|
`coursebank validate` checks that every file is correct and consistent, and it exits 2 if it finds any problem.
|
|
It covers the course (empty identity fields, grade scale, duplicate ids, reference fields, prerequisite cycles, level ceilings), every bank against the course (the rules above, unknown targets, lectures, stimuli, and references), every assessment record against the catalog, every seal's stem digests, and calibration against the banks.
|
|
For the 2.1 registries it checks id prefixes, ids unique across every registry, the entries each target lists, equation symbols, values, citation keys, brace references in student-facing prose, that every `conflicts` and `target_text_forbid` pattern compiles, and each option's `misconception_id`.
|
|
A bank item that carries a `calibration:` block fails validation.
|
|
|
|
`coursebank lint` checks craft.
|
|
It has 44 rules in six families, and each rule has a code.
|
|
|
|
| Family | Codes | What they flag |
|
|
|--------|-------|----------------|
|
|
| Cueing (9) | `cue-key-longest`, `cue-uneven-length`, `cue-word-repeat`, `cue-all-of-the-above`, `cue-absolute`, `cue-overlap`, `cue-near-duplicate`, `cue-key-position`, `cue-option-count` | features that let a test-taker guess without knowing |
|
|
| Clarity (6) | `clarity-no-task`, `clarity-stem-length`, `clarity-negation`, `clarity-vague`, `clarity-reading-load`, `clarity-unfocused` | stems that do not pose a clear task |
|
|
| Completeness (7) | `complete-distractor-design`, `complete-distractor-explanation`, `complete-student-feedback`, `complete-alt-text`, `complete-format-level`, `complete-bonus-policy`, `complete-design-conflict` | missing design intent, feedback, or alt text |
|
|
| Pool (3) | `pool-thin`, `pool-unused`, `pool-unjustified-retirement` | too few usable distractors for a form |
|
|
| Evidence (5) | `evidence-nonfunctioning`, `evidence-stale`, `evidence-difficulty`, `evidence-discrimination`, `evidence-duplicate-stem` | observed results that contradict the prediction or the text |
|
|
| Glossary (14) | `glossary-unused`, `glossary-no-textbook-name`, `glossary-no-exam-wording`, `glossary-circular-definition`, `glossary-coined-in-item`, `glossary-value-unsourced`, `glossary-value-conflict`, `glossary-forbidden-target-wording`, `glossary-unlisted-term`, `glossary-recall-value-printed`, `glossary-equation-no-implications`, `glossary-unresolved-reference`, `glossary-defined-elsewhere`, `glossary-proposed` | entries nothing needs, coined terms in items, printed recall values, conflicting wordings of a number, and entries waiting for approval |
|
|
|
|
Each finding has a severity of note, low, medium, or high, and a finding's severity can differ from its rule's nominal one (for example, `cue-uneven-length` is medium when the spread is extreme).
|
|
Two rules report notes: `glossary-defined-elsewhere` and `glossary-proposed`.
|
|
Retired items are skipped, and findings are grouped by item.
|
|
The options are `--list-rules` (list every code and severity, then exit), `--only CODES`, `--ignore CODES` (both comma-separated), `--min-severity note|low|medium|high` (default `note`), and `--no-fail`.
|
|
`lint` exits 2 if it finds anything above a note, unless `--no-fail` is given, which is why a pre-commit hook of `coursebank validate && coursebank lint` works.
|
|
To block a commit only on serious findings, add `--min-severity high`.
|
|
|
|
`coursebank catalog` summarizes the pool: item, bank, and assemblable counts, and counts by status and by level.
|
|
`catalog --coverage` adds a table per objective and per target (items, ready items, targets, maximum level) and lists gaps, exiting 2 if there are any.
|
|
The gap kinds are an uncovered objective, an objective with no approved items, an objective with a single assemblable item, an objective covered only at level 1 despite a higher ceiling, a lecture with no assessed objective, and an item with no objective.
|
|
|
|
## Assembling an exam
|
|
|
|
`assemble` draws a new assessment from the pool and writes it to `assessments/ID.yaml`.
|
|
It takes a blueprint on the command line:
|
|
|
|
| Flag | Meaning | Default |
|
|
|------|---------|---------|
|
|
| `--title TITLE` | assessment title | the id |
|
|
| `--kind` | `exam`, `quiz`, `homework`, `practice`, or `final` | `exam` |
|
|
| `--date YYYY-MM-DD` | the date, which also anchors the cooldown | today |
|
|
| `--platform` | `paper`, `canvas`, or `other` | `paper` |
|
|
| `--levels L=N,...` | scored items per level, for example `1=6,2=8,3=10,4=6` | none; `assemble` fails if no scored items are requested |
|
|
| `--bonus L=N,...` | bonus items per level | none |
|
|
| `--require OBJECTIVE=N,...` | at least N items for an objective, satisfied by items on any of its targets | none |
|
|
| `--lectures`, `--topics`, `--banks` | restrict the draw (comma-separated; the filters intersect) | no restriction |
|
|
| `--max-per-bank N` | at most N items from any one bank | no limit |
|
|
| `--cooldown DAYS` | avoid items used within this many days | 180 |
|
|
| `--seed N` | seed for the draw | 0 |
|
|
| `--forms N` | number of forms | 1 |
|
|
| `--dry-run` | show the draw, write nothing | off |
|
|
| `--force` | overwrite an existing record | off |
|
|
|
|
`assemble` fails with a message if no scored items are requested.
|
|
It has no `--term` flag: the record takes the course term.
|
|
It refuses to overwrite an existing record unless you pass `--force`, because replacing an administered assessment invalidates stored responses.
|
|
|
|
### How the draw works
|
|
|
|
1. The pool: only approved, unretired items are eligible, narrowed by the bank, lecture (an item's `sources`), and topic filters. An empty pool is an error.
|
|
2. Objective minimums first: for each `--require` entry, `assemble` picks that many non-bonus items tagged to the objective or to one of its targets, respecting the per-bank cap and the cooldown. A shortfall is a note, not a failure.
|
|
3. Level quotas: for each level in `--levels`, in ascending order, it ranks the remaining candidates and fills the quota. The first pass honors the cooldown and the per-bank cap. A second pass relaxes the cooldown when the first falls short, and adds a note such as "reused ... inside the 180-day cooldown". `assemble` fails when items are still short, and says how many are blocked by the cooldown and by the cap.
|
|
4. Bonus: the same fill runs over items flagged `bonus: true`.
|
|
5. Ordering: scored items are sorted by level and then id so that the form ramps in difficulty, and bonus items are numbered after the scored ones.
|
|
|
|
Items placed for an objective minimum are not counted against the level quotas, so the scored total can exceed the sum of the `--levels` counts.
|
|
|
|
The ranking is deterministic.
|
|
Candidates are shuffled with a generator seeded from the seed and a label, then stable-sorted so that never-used items come first, then the item whose last use is oldest, then the item with the fewest uses.
|
|
The shuffle is the tie-break.
|
|
No system entropy is involved, so the same seed, the same pool, and the same history give the same exam.
|
|
|
|
Two changes illustrate the effect.
|
|
Lowering `--cooldown` from 180 to 30 stops the first pass from blocking items last used between 30 and 180 days ago, so `assemble` is less likely to need the second pass or to fail with a count of items blocked by the cooldown. The ranking still prefers never-used items.
|
|
Changing `--seed` changes the tie-breaks among equally fresh items and leaves the freshness ordering alone.
|
|
|
|
### What the record stores
|
|
|
|
For each placement the record stores the question `number`, the item id, the stem digest, the option-set `variant` digest, the fingerprint, the points, whether it is a bonus item, the `key` (option ids keyed), the `distractors` (option ids offered), the level, and the learning targets.
|
|
It also stores the course term, the blueprint, and the forms.
|
|
The options shown are drawn once, by `assemble`, and written to the record.
|
|
They are never sampled at export, so every export of the same record shows the same options.
|
|
The draw does not fail when an item's pool is too short for a full form; the form is short and `lint` reports `pool-thin`.
|
|
|
|
Placement `number` is the printed question number and the join key to grading exports.
|
|
Numbers must run 1 to N without gaps.
|
|
Option order on paper is computed from the form's seed and is never stored, so the paper, the key, and the answer sheet agree.
|
|
`assemble` writes forms with `shuffle_items: false` and `shuffle_options: true`, and gives each form a seed derived from `"{assessment id}/form-{label}"`.
|
|
Form labels run A, B, and on to Z, then AA.
|
|
To give a form a different arrangement, edit the `seed:` in the record, re-export, and re-seal with `--force` (allowed only for an assessment that was never administered).
|
|
|
|
Corrections made after the fact go in the record: `credit_overrides` (a map from option to a credit between 0 and 1) and `dropped: true` with `dropped_as: removed` or `full_credit`.
|
|
`dropped` and `credit_overrides` cannot be combined on one placement.
|
|
|
|
### Inspecting a record
|
|
|
|
`assessment new ID` writes an empty record (default `--kind exam`) for a hand-built assessment, and `validate` reports that it lists no items until you add some.
|
|
`assessment list` shows ID, kind, date, item count, and points for every record.
|
|
`assessment show ID` prints the by-level counts and, for every question, its number, id, key, points, and estimated working minutes, then validates the record and exits 2 if there are problems.
|
|
|
|
The working-minutes estimate uses these per-level values when an item has no `design.expected_time_seconds`:
|
|
|
|
| Level | Seconds |
|
|
|-------|---------|
|
|
| level 1 | 35 |
|
|
| level 2 | 65 |
|
|
| level 3 | 95 |
|
|
| level 4 | 130 |
|
|
| level 5 | 165 |
|
|
|
|
An assessment is found by its file name or by the `id:` inside the record.
|
|
In both course repositories the ids are the `id:` values (`A1.2`, `e1`), while the file names are lowercase with hyphens (`a-1-2.yaml`), and everything coursebank does is keyed on the id.
|
|
|
|
## Reuse history
|
|
|
|
Reuse is derived from the records and never stored.
|
|
`usage history` lists, for every item that has been used, when and where (as `assessment#number`) and the last-used date.
|
|
Name an item (`usage history ITEM`) to see one.
|
|
`usage unused` lists approved items that no record has ever placed.
|
|
`assemble` reads the same history, so the cooldown and the preference for never-used items rest on the records you can read in a diff.
|
|
The history spans every assessment record, of any kind and any term.
|
|
Deleting an assessment record removes its uses from the history, so keep records that were administered.
|
|
|
|
## Sealing before printing
|
|
|
|
`seal ID` freezes an administration: what the students held in their hands.
|
|
The seal stores a copy of each stem and option text as administered, the printed-letter-to-option map for every form and every question, and SHA-256 digests per item, per form, and for the whole file.
|
|
It writes `seals/ID.yaml`.
|
|
The purpose is practical.
|
|
A paper exam shuffles options, so a Gradescope export names letters as printed on that form, and only the seal can translate a printed `C` on form B back to the option it was.
|
|
Ingest derives the map from the record as it stands today and warns when no seal exists, and that map is wrong if the bank or record has changed since printing.
|
|
|
|
| Flag | Effect |
|
|
|------|--------|
|
|
| `--check` | compare the existing seal with the live course, print findings with severity, and exit 2 on drift |
|
|
| `--force` | overwrite an existing seal; use it only for an assessment that was never administered |
|
|
| `--form A,B` | seal only these forms (default: every declared form) |
|
|
| `--no-content` | store fingerprints and digests only, not stem and option text |
|
|
| `--out PATH` | write somewhere other than `seals/ID.yaml` |
|
|
|
|
A seal is meant to be written once, before the exam is printed.
|
|
`seal --check` reports drift with a code and a severity, and only `high` severity blocks.
|
|
High-severity codes include a changed whole-file or form digest, a removed, replaced, or edited item, a changed key, a changed option count, a removed form, and a changed option permutation.
|
|
Medium covers changed points and an added item, and low covers retagged objectives and a changed level.
|
|
The command prints per-form counts and advisories about key balance after writing a seal.
|
|
|
|
A seal also records dropped questions (the e1 seal of one course repository lists `dropped: [31, 35]`), and when `migrate counters` rewrites an item id, the old file digest is appended to `superseded_digests`.
|
|
`validate` checks every seal's stem digests against the bank, so a reworded stem is caught even long after the exam.
|
|
|
|
The digests come in three kinds.
|
|
An item's fingerprint is a hash of its trimmed stem, each option (sorted by id) as `id|correct|text`, and the stimulus id, using FNV-1a, which is not a security primitive.
|
|
The stem digest covers the stem and stimulus only, and the variant digest covers the stem digest plus the options as administered and which were keyed; the variant digest is the key under which statistics are pooled.
|
|
Seals use SHA-256 and prefix digests with `sha256:`.
|
|
|
|
Canvas ingest does not use the seal, because a Canvas export names answer text and not printed letters, and the homework in the two course repositories has no seals.
|
|
|
|
## Exporting
|
|
|
|
Every export reads the assessment record and the bank, so a re-export of the same record gives the same documents.
|
|
|
|
### Typst: paper exam, key, and answer sheet
|
|
|
|
`export typst ID` writes Typst source, one file per form and variant, into `build/` by default:
|
|
|
|
- `ID-FORM.typ` (the exam), `ID-FORM-key.typ`, and `ID-FORM-answer-sheet.typ` (a bubble sheet sized from each item's option count).
|
|
- `--form` takes one form id (default `A`) or `all`. The option takes a single value, whatever the help text's "repeat or pass all" suggests.
|
|
- `--variant` selects documents. The default set is `exam`, `key`, `answer-sheet`.
|
|
- `--out DIR` changes the directory, `--json` also writes the payload as JSON, and `--dry-run` prints the question count and the resolved template and writes nothing.
|
|
- `--template FILE` is accepted only when exactly one `--variant` is given.
|
|
|
|
The parser also accepts the tokens `student-report` and `cohort-report`. Those documents are built from responses, not from an assessment record, so `export typst` has nothing to build them from [CHECK: the code comment says a diagnostic cannot be built from an assessment record, but what `export typst --variant student-report` actually does was not run]. Use `report students --typst` and `report cohort --typst` for them.
|
|
|
|
The tool writes Typst source and does not compile it.
|
|
Compile with `typst compile` (in the pixi docs environment, `pixi run -e docs typst compile FILE.typ`).
|
|
`export typst` exits 2 when it printed an advisory warning: an unbalanced `[` or `]` in a stem or option, a template with no markers, or a chemical formula in a slot when the template lacks a whalogen import.
|
|
|
|
**You own the layout.** In the words of `docs/TYPST.md`, the tool does not write Typst documents: it loads a Typst file you own, finds the markers in it, and injects data.
|
|
The markers are line comments, `// coursebank:begin SLOT` and `// coursebank:end SLOT`, and the text between them is replaced.
|
|
The markers define three slots: `meta` (a `#let cb-meta = (...)` binding), `questions` (one `#render-question((...))` call per printed item), and `data` (a `#let cb-data = (...)` binding with the metadata and the questions).
|
|
The marker lines survive, so an exported document can be reused as a template.
|
|
Malformed markers (an unknown slot, an unclosed region, a stray end, a nested region, a slot claimed twice) are all reported at once.
|
|
|
|
The template is found in this order: a `--template` path or `template:` in the render config, then `templates/ID-VARIANT.typ`, then `templates/VARIANT.typ`, then the built-in template compiled into the binary.
|
|
The five built-in templates are `exam.typ`, `key.typ`, `answer-sheet.typ`, `student-report.typ`, and `cohort-report.typ`.
|
|
`template list` shows the lookup chain, which entry resolved, and the slots each template declares.
|
|
`template dump` writes the built-in templates into `templates/` to edit (all five by default, `--variant` to choose, `--force` to overwrite, `--stdout` to print), and it exits 2 if it skipped an existing file.
|
|
`init` does not create `templates/`; `template dump` does.
|
|
|
|
`templates/typst.yaml` configures the payload in three layers: built-in defaults for the variant, then `defaults:`, then `variants:`.
|
|
`template config` writes a starter file and `template config --resolved VARIANT` prints the fully resolved settings.
|
|
The keys include `question_fn`, `content` (`content` or `str`), `letters` (`upper`, `lower`, `numeric`, `roman`, `nothing`), `reveal`, `stimulus`, `fields`, `number_from_record`, and a free-form `extra`.
|
|
Unknown keys are rejected.
|
|
|
|
`reveal` decides whether the payload contains the answer at all.
|
|
Its values are `nothing`, `key`, and `everything`, and for the exam the correct-answer field is absent from the payload.
|
|
The built-in exam and student-report variants use `nothing`, and the built-in key and cohort-report variants use `everything` [CHECK: the sheets do not say what `key` includes that `nothing` leaves out, or what `everything` adds to `key`; confirm against docs/TYPST.md].
|
|
Do not set `reveal: key` on the exam to make a solutions copy: export the `key` variant instead.
|
|
Because option order comes from the form's recorded seed and question numbers come from the record, the paper, key, and answer sheet always agree.
|
|
The full template contract is in `docs/TYPST.md`.
|
|
|
|
### QTI: Canvas
|
|
|
|
`export qti ID` builds a QTI 1.2 package for Canvas in the common-cartridge style, written to `build/ID-FORM.zip`.
|
|
Import it in Canvas under Settings, Import Course Content, QTI .zip file.
|
|
The flags are `--form` (default `A`), `--out FILE`, `--no-feedback` (leave out per-option feedback), and `--attempts N`.
|
|
The item types map as follows.
|
|
|
|
| Item format | QTI type |
|
|
|-------------|----------|
|
|
| `single_best_answer` | `multiple_choice_question` |
|
|
| `multiple_response` | `multiple_answers_question` (full credit only for the exact set) |
|
|
| `true_false` | `true_false_question` |
|
|
| `open_response` | `essay_question` |
|
|
|
|
The essay mapping has not been round-tripped through a live Canvas import in this build, according to a comment in the code.
|
|
A choice item with no keyed option aborts the export and lists the problems.
|
|
Ids are derived by hashing the assessment id and item id, so a re-export keeps identical ids.
|
|
Inline math is rewritten to `\( ... \)` because Canvas's MathJax does not read bare dollar signs.
|
|
The attempt limit, the scoring policy, and shuffling come from the assessment record (`attempts`, default -1 for unlimited), not from the command line.
|
|
The command accepts `--attempts` and ignores it.
|
|
Figures from item `assets` are not included in the package.
|
|
|
|
### Practice worksheet and solutions
|
|
|
|
`export practice ID` writes Quarto Markdown: `ID-FORM.qmd` (stems and options only) and `ID-FORM-solutions.qmd` (the key, worked reasoning, per-option notes, the rubric, and review citations).
|
|
`--variant worksheet|solutions` selects one (both by default), `--out DIR` sets the directory (default `build/`), and `--no-answer-space` drops the written-answer space after open-response questions.
|
|
Render with `quarto render FILE.qmd`.
|
|
|
|
### Site: gated solutions for a web page
|
|
|
|
`export site ID` writes a Quarto partial and an encrypted solutions bundle for a password-gated homework page.
|
|
It writes `_questions.qmd` and `ID-solutions.json` to `--out` (default `build/`).
|
|
The partial contains the questions and an empty hidden slot per question, with no answers in the file; include it from a page with `{{< include _questions.qmd >}}`.
|
|
`--assets DIR` also writes `questions.css` and `solutions.js`, which a Quarto site adds through `_quarto.yml`.
|
|
In the browser, `solutions.js` derives the key from the password, decrypts every solution, and re-typesets the math.
|
|
It needs a secure context, because it uses the Web Crypto application programming interface (API).
|
|
|
|
The bundle is encrypted with a key derived by PBKDF2-HMAC-SHA256 (250,000 iterations), and each solution is sealed with AES-256-GCM under its own random 96-bit initialization vector.
|
|
The command generates a password unless you pass `--password`.
|
|
A generated password is 16 characters from the alphabet `0123456789abcdefghjkmnpqrstvwxyz`, shown in groups of four (80 bits), from the system random generator.
|
|
The command prints the password and stores it nowhere, so write it down when it appears.
|
|
The guide describes the lock as a deterrent, not a grading system of record (`docs/guide/assignment.md`).
|
|
|
|
### Markdown review copy
|
|
|
|
`export md ID` writes the items as Markdown for a reviewer, to `build/ID.md`.
|
|
`--with-key` adds correct-option marks, the rationale, and the option explanations.
|
|
|
|
### Lecture and reference outputs
|
|
|
|
`lecture readings ID` and `lecture objectives ID` write the readings block and the learning-objectives block for one lecture, as Quarto Pandoc definition lists (`--style quarto`, the default) or as plain Markdown bullets (`--style plain`).
|
|
They print to standard output without `--out`.
|
|
A lecture's pages leave off its prerequisites: an objective or target that names the lecture in its `lectures:` list but was covered in an earlier lecture is not printed, unless the lecture lists its objective in `teaches`.
|
|
Prose that cites a prerequisite, such as `{t-control-at-far-steps}`, renders as its number on the page of the lecture that teaches it.
|
|
`lecture page ID` writes the whole page for one lecture: its description, each objective with its targets (each with its explanation and its ceiling named by `policy.levels`), new and earlier terms, equations, numbers to know, common mistakes, readings with minutes, and materials.
|
|
`lecture glossary` writes the course glossary page: every term alphabetically, then every equation and value, each with the lecture that introduces it.
|
|
Both take `--style` and `--out`, and `lecture page` takes `--glossary-href` for where links to earlier terms point (default `glossary.html`).
|
|
None of these prints `authoring`, `notes_private`, `evidence`, `conflicts`, or `status`.
|
|
`references export` writes the bibliography as `hayagriva` (the default, read by Typst), `csl-json` (read by Zotero and Pandoc), or `bibtex`.
|
|
The `label`, `role`, `base_url`, and `note` fields do not survive into those formats.
|
|
|
|
## Ingesting grading data
|
|
|
|
Two grading exports are supported, and the assessment id is required for both.
|
|
|
|
| Command | Reads |
|
|
|---------|-------|
|
|
| `ingest gradescope SOURCE...` | one directory of per-question CSVs per form, files named `1.csv` to `N.csv`, or `N_Title.csv` for a titled question |
|
|
| `ingest canvas FILE` | the Canvas "Student Analysis" quiz CSV (not the gradebook, not the Item Analysis) |
|
|
|
|
Both readers produce the same long-format row, one per student per item.
|
|
No other platform is implemented, even though the `platform` field can say `paper`, `canvas`, or `other`.
|
|
|
|
The flags shared by both commands are:
|
|
|
|
| Flag | Meaning |
|
|
|------|---------|
|
|
| `--assessment ID` | the assessment record the responses belong to (required) |
|
|
| `--date YYYY-MM-DD` | the administration date (default: the record's date) |
|
|
| `--form ID` | which form, when forms were used |
|
|
| `--pseudonymize` | replace student identifiers with keyed pseudonyms; requires `--salt-file` |
|
|
| `--salt-file PATH` | the secret for the keyed hash; at least 16 bytes once whitespace is stripped |
|
|
| `--format parquet\|csv` | storage format (default Parquet) |
|
|
| `--dry-run` | parse and report, write nothing |
|
|
|
|
`ingest gradescope` adds `--allow-mismatch` and `--recorded-numbers`.
|
|
|
|
### Several forms
|
|
|
|
Write each Gradescope source as `FORM=DIR`, for example `A=exports/e1-a B=exports/e1-b`.
|
|
A bare path takes its form from `--form`, and giving neither is an error.
|
|
A form given twice, or a form the record does not declare, is an error.
|
|
Gradescope numbers questions by printed position, and the decoder maps each printed position to the recorded question number and each printed letter to the option it was, using the seal.
|
|
The decoded rows keep the printed letters in `selected` and `eliminated` and add the bank-lettering in `selected_source` and `eliminated_source`.
|
|
|
|
Three checks guard against a mislabeled directory or a bad key.
|
|
The first compares each directory's graded key with every declared form's printed key, so that swapped directories are detected; a mismatch blocks the ingest unless you pass `--allow-mismatch`, which turns it into a warning.
|
|
The second flags a student who appears on more than one form, and a form that is missing a question the record expects.
|
|
The third warns when two forms differ in mean score by 5 percentage points or more with at least 5 students on each.
|
|
`--recorded-numbers` is for an export that already uses recorded question numbers.
|
|
Ingest also warns when a distractor received partial credit at grading time, and the analysis then flags the item as `ambiguous`.
|
|
Ingest uses a seal when one exists and prints any drift as warnings.
|
|
|
|
### What the readers do
|
|
|
|
The Gradescope reader takes the keyed options from the rubric columns whose point value equals the top value, treats `true` in a rubric cell as a selection, and computes `correct` as whether the chosen set equals the keyed set.
|
|
A blank row has `correct` empty.
|
|
`response_time_seconds` is always empty for Gradescope.
|
|
The Canvas reader works from answer text, because Canvas exports text and not letters.
|
|
It recovers the letter by matching normalized answer text against the item's option text, and an answer that does not match keeps its score, loses its letter, and triggers a warning.
|
|
It aligns questions to item numbers by matching stem text against the record, falling back to record order with a warning.
|
|
Canvas ingest does not use the seal, so there is no printed-letter translation.
|
|
|
|
### Privacy
|
|
|
|
The tool reads student names from the export and does not store them: names are not part of the stored schema, so reports fall back to the student key.
|
|
Without `--pseudonymize`, the stored rows still carry the institutional id, email, and section.
|
|
With `--pseudonymize`, the student key becomes `s-` followed by 12 hexadecimal characters of an HMAC-SHA-256 of the lowercased, trimmed id (the institutional id, else the email, else the existing key), keyed with the salt, and the `sid`, name, and email are cleared.
|
|
coursebank refuses `--pseudonymize` without a salt file, so a file never looks anonymous when it is not.
|
|
Keep the salt outside every repository.
|
|
The error message suggests generating it with `openssl rand -hex 32 > PATH`, where `PATH` is a location outside the repo.
|
|
According to the course scripts, losing the salt exposes nobody; it only ends the ability to match a student across administrations.
|
|
`init`'s `.gitignore` excludes `*.salt` and `.coursebank-salt`, and `data/` is ignored as a whole.
|
|
|
|
### What gets stored
|
|
|
|
Ingest prints the number of responses, students, and items and writes one file per administration into `data/`.
|
|
The administration id is `COURSE/TERM/ASSESSMENT`, using the course code exactly as written, and the file name is that id sanitized: lowercased, with letters, digits, `.`, and `-` kept and every other run replaced by `_`.
|
|
For example, with the course code `BIOSC 1540`, the administration `BIOSC 1540/2026s/exam-4` is stored as `biosc_1540_2026s_exam-4.parquet`.
|
|
Writing replaces an existing file for that administration and does not append, so re-ingesting after a correction is safe.
|
|
`coursebank data` lists what is stored, with rows, students, items, and file.
|
|
|
|
The Parquet file is Snappy-compressed, with every column declared non-nullable and an empty string or zero used where a value is unknown.
|
|
The columns, in order, are:
|
|
|
|
- identity and context (text): `administration_id`, `course`, `term`, `assessment_id`, `date`, `form`
|
|
- student (text): `student_key`, `sid`, `email`, `section`
|
|
- the item: `item_number` (the recorded question number and the join key to the record), `item_ref` (the item id), `item_version`, `variant` (the option-set digest as administered)
|
|
- the response: `selected`, `eliminated` (comma-joined letters as printed on that student's form), `correct` (`1`, `0`, or empty), `credit` (fraction of the item's points), `points_possible`, `score`, `response_time_seconds`
|
|
- item metadata copied from the record and bank: `level`, `learning_objectives` (comma-joined target ids; the column keeps its old name), `topics`, `bonus`, `dropped`
|
|
- the decoded form: `form_position`, `dropped_full_credit`, `selected_source`, `eliminated_source`
|
|
|
|
On read, `variant`, `form_position`, `dropped_full_credit`, `selected_source`, and `eliminated_source` are optional, so older files still load.
|
|
A file with other missing or mistyped columns is rejected with a message saying it was probably not written by coursebank.
|
|
Reading accepts either Parquet or CSV regardless of the format that wrote the file.
|
|
|
|
## Analysis and calibration
|
|
|
|
### Classical test theory
|
|
|
|
`analyze items ID` runs a classical item analysis on one administration.
|
|
`--pooled` reads every stored administration of that assessment id across terms, and without it only the current term is read.
|
|
Only scored, undropped, non-bonus items enter the analysis.
|
|
For each question it reports:
|
|
|
|
- Difficulty, p: the mean of the 0/1 correct column over all examinees, counting a blank as incorrect. The blank rate is reported separately. Partial credit is coded as incorrect for these statistics.
|
|
- Discrimination, r: the corrected point-biserial, the Pearson correlation between the item's 0/1 column and the total score on the other items. It is undefined when either column has no variance.
|
|
- Discrimination index, D: the proportion correct in the top group minus the proportion correct in the bottom group, where each group is the top or bottom 27% of students by total score. It is omitted for fewer than 6 students or when the groups overlap.
|
|
- Distractor analysis: for each option, the selection rate, the option's point-biserial, and its rate in the upper and lower groups. Every declared option appears even if nobody chose it.
|
|
- Reliability: KR-20, the standard error of measurement, the mean p and the mean point-biserial, with a plain-language reading of alpha in bands from very low to very high (0.7 to 0.8 is read as acceptable for a classroom exam) and a note when the exam has fewer than 20 items.
|
|
- Flags and the revise queue, described next.
|
|
|
|
A sample smaller than 100 examinees triggers a small-sample warning that quotes a standard error of `1/sqrt(n-1)`.
|
|
|
|
The analysis defines 11 flags.
|
|
The table shows the defaults for the flags that use a threshold.
|
|
|
|
| Flag | Fires when |
|
|
|------|-----------|
|
|
| `negative_discrimination` | point-biserial below -0.05 |
|
|
| `low_discrimination` | point-biserial below 0.15 (and not already negative) |
|
|
| `too_easy` | p above 0.95 |
|
|
| `too_hard` | p below 0.25 and the point-biserial is not at least 0.15 |
|
|
| `distractor_outperforms_key` | a distractor's point-biserial exceeds the key's and it is chosen by at least 10% |
|
|
| `key_underperforms` | a distractor's upper-group rate exceeds the key's and is at least 0.25 |
|
|
| `nonfunctioning_distractor` | a distractor is chosen by 5% or fewer |
|
|
| `ambiguous` | a non-key option earned partial credit at grading |
|
|
| `design_mismatch` | observed p is more than 0.25 from the predicted difficulty, or the point-biserial is outside the predicted band, on a calibration with at least 50 examinees whose fingerprint matches the item |
|
|
| `high_rapid_guess` | see the limits section; cannot fire today |
|
|
| `dif_flagged` | see the limits section; cannot fire today |
|
|
|
|
Four flags block: `negative_discrimination`, `distractor_outperforms_key`, `ambiguous`, and `key_underperforms`.
|
|
The revise queue lists flagged items with blocking flags first, then by lowest point-biserial, and `analyze items` prints the worst ten and exits 2 if the queue is not empty.
|
|
A script can branch on that exit code to review the revise queue.
|
|
|
|
The bands for `design.expected_discrimination` are:
|
|
|
|
| Band | Point-biserial |
|
|
|------|----------------|
|
|
| low | below 0.20 |
|
|
| moderate | 0.15 to 0.40 |
|
|
| high | 0.30 or above |
|
|
|
|
`--record` also writes the aggregates under `analysis/administrations/`, described below.
|
|
|
|
### Item response theory
|
|
|
|
`analyze irt ID` fits a one-parameter, two-parameter, or three-parameter logistic model by marginal maximum likelihood.
|
|
`--model` takes `rasch`, `two-pl` (the default), or `three-pl`.
|
|
The fit uses 41 quadrature points over a standard normal ability range of -4 to 4, and it estimates a discrimination `a`, a difficulty `b`, and (for 3PL) a guessing parameter `c` per item, with standard errors, plus an expected-a-posteriori ability for each examinee.
|
|
Priors are on by default (`a` log-normal with a log standard deviation of 0.5, `b` normal with a standard deviation of 2), because unpenalized maximum likelihood has no finite solution for an item that everyone answered correctly.
|
|
`--no-priors` turns them off and warns that divergence is likely on a single class.
|
|
Warnings appear when n is below 100 for the Rasch model, 200 for the two-parameter model (2PL), and 1000 for the three-parameter model (3PL).
|
|
Responses are dichotomous, so partial credit counts as incorrect.
|
|
|
|
### Per-student mastery
|
|
|
|
`analyze students ID` reports a roster, the objectives the class did not meet, and archetype patterns from a seeded k-means (k = 3) over per-level rates, which runs only with at least 9 students and two levels.
|
|
A student's mastery of an objective is `Meeting` when the rate is at least `mastery_threshold`, `Developing` when it is at least 0.6 of the threshold, `NotYet` otherwise, and `NotEnoughEvidence` when fewer than `min_items_for_mastery` items bear on the objective.
|
|
Objectives are classified; targets are reported as itemized evidence.
|
|
A rate is marked confident when the lower bound of its 95% Wilson interval clears the threshold, so the same rate over more items can be confident when it is not over fewer.
|
|
|
|
### Recording an administration
|
|
|
|
`analyze items ID --record` writes two CSV files under `analysis/administrations/`, named for the slug of the administration id:
|
|
|
|
- `...-items.csv`: one row per question (administration id, assessment, term, date, forms, number of examinees, the coursebank version, item, number, variant, stem digest, n, p, point-biserial, discrimination index).
|
|
- `...-options.csv`: one row per question and option (item, number, option, keyed, selection rate, point-biserial, upper and lower group rates).
|
|
|
|
They are cohort aggregates, and the readers reject any header outside a fixed list, so a student column cannot slip in.
|
|
They are meant to be committed, and they are written once and never overwritten.
|
|
They carry no IRT parameters.
|
|
|
|
### Calibration
|
|
|
|
`calibrate` pools the statistics from every stored administration and writes them back for the next assembly.
|
|
It takes no assessment argument.
|
|
|
|
| Flag | Effect |
|
|
|------|--------|
|
|
| `--apply` | write `analysis/calibration.yaml`; without it, print the plan and write nothing |
|
|
| `--no-irt` | skip the IRT fit |
|
|
| `--include-practice` | include practice assessments (excluded by default) |
|
|
| `--min-n N` | minimum pooled examinees before an item is written (default 10) |
|
|
|
|
Each administration is analyzed on its own, so the point-biserial is measured against that administration's test, and the results are pooled per item.
|
|
The pooled difficulty is the examinee-weighted average.
|
|
The point-biserial and the discrimination index are examinee-weighted averages of the within-administration values, not recomputed from merged data.
|
|
Flags are the union.
|
|
Statistics are also kept per option set (the variant digest), so a stem that appeared with different options produces separate `variants` records.
|
|
The plan prints a diff of what would change for each item, and tags items as new or as computed from different text.
|
|
`calibrate --apply` rewrites only `analysis/calibration.yaml`, and each entry stores the fingerprint it was computed against.
|
|
Every pass recomputes from all stored administrations, rather than accumulating deltas.
|
|
|
|
Pooled IRT is narrower than the other statistics.
|
|
The IRT fit uses a single administration, the one with the most students-by-items cells, and it needs at least `--min-n` students and 5 items in it.
|
|
A warning says that the parameters come from one administration, because linking across administrations is not attempted.
|
|
Read `irt` in the calibration file as describing that one administration.
|
|
|
|
### Where results are stored
|
|
|
|
| Where | What | Git |
|
|
|-------|------|-----|
|
|
| `data/` | per-student responses | not committed |
|
|
| `analysis/` | cohort aggregates: `calibration.yaml` and `administrations/*.csv` | committed |
|
|
| `reports/` | per-student and cohort reports | not committed (per-student files carry student keys) |
|
|
|
|
### Two cases that show how the pieces interact
|
|
|
|
Two cases show how changing one thing changes the outcome.
|
|
|
|
- Retagging versus rewording: move an item from one learning target to another and its fingerprint is unchanged, so its pooled statistics stay valid and `evidence-stale` does not fire. Reword its stem and the fingerprint changes, the next `calibrate` plan tags the item as computed from different text, and `evidence-stale` fires at high severity until it is recalibrated. In an assessment record that already uses the item, a reworded stem is a validation error: give the new wording a new id and record `supersedes`.
|
|
- Prediction, observation, and sample size: the `design` block predicts a difficulty and a discrimination band. `evidence-difficulty` fires when the observed difficulty is 0.25 or more from the prediction, and `evidence-discrimination` fires when the observed point-biserial falls outside the predicted band. The analysis flag `design_mismatch` adds a sample-size condition: it needs a calibration of at least 50 examinees whose fingerprint matches the item's current fingerprint, and below that a miss is recorded in the prediction notes only. The old calibration no longer matches the fingerprint after a rewording, so it cannot raise the flag until the item is recalibrated.
|
|
|
|
## Reports
|
|
|
|
`report students ID` writes a diagnostic for each student, and `report cohort ID` writes an instructor item analysis.
|
|
Both default to Markdown, and each can also write HTML (`--html`), Typst source (`--typst`, to compile with `typst compile`), and the payload as JSON (`--json`).
|
|
`--no-markdown` suppresses the Markdown, and `--template FILE` uses your own Typst template.
|
|
|
|
The student report answers "what should I do next?"
|
|
It includes the student's performance by level and by objective, strengths, the objectives to focus on, per-question results with the feedback and hint written for the option the student chose, dropped questions, lectures to review with readings, and a study list.
|
|
It leaves out the correct answers, other students' data, and any numeric rank.
|
|
Two optional switches change that: `--solutions` adds the worked solution for every missed question (the command warns about the effect on reusing the exam), and `--misconceptions` names the misconception each chosen distractor was written to catch, which is off by default.
|
|
Each missed target shows what meeting it looks like, from the target's `explanation`.
|
|
A chosen option with a `misconception_id` shows that misconception's correction, and misses that share one across questions are grouped under it.
|
|
`--ability` adds the IRT ability estimate and `--no-comparison` omits the comparison with the class.
|
|
Section switches turn individual sections off: `--no-questions`, `--no-feedback`, `--no-hints`, `--no-levels`, `--no-objectives`, `--no-strengths`, `--no-focus`, `--no-dropped-questions`, `--no-review-lectures`, and `--no-study`.
|
|
Files go to `reports/ID/`, named for the student key, unless `--out DIR` says otherwise.
|
|
The report is routed by the key, because the stored schema has no names.
|
|
A pseudonymized ingest produces reports that cannot be routed to a student, which is why the two course repositories ingest with ids first, compile the reports, and then re-ingest pseudonymized (see the worked pictures).
|
|
|
|
The cohort report answers "what should I fix?"
|
|
It holds the score distribution (mean, median, standard deviation, minimum, maximum, and ten bins), reliability, per-level and per-objective rows, objective gaps below the mastery threshold, letter-grade bins when `policy.grade_scale` is set, per-lecture and per-question rows, dropped-question detail, a triage for each flagged question, a comparison of predictions with results, per-form rows, and warnings.
|
|
The triage buckets are discard (the point-biserial is below -0.05, or below 0.05 with p near chance), rekey (a distractor chosen by at least 20% with a positive point-biserial at least equal to the key's), bounded (a weak question that is too easy or too hard), revise (weak, or with a nonfunctioning option, or ambiguous), and reteach (a question with a point-biserial of at least 0.20 whose difficulty p is below the course mastery threshold).
|
|
The default path is `reports/ID-cohort.md`, and the command exits 2 when the revise list is not empty.
|
|
`report students` prints a drift warning to the error stream if the bank has moved since the seal.
|
|
|
|
## Lectures, references, and site content
|
|
|
|
`lecture coverage [--lecture ID]` lists the readings behind each learning target and which targets have none.
|
|
It exits 2 when an assessed target has no reading.
|
|
`references list` prints every work with its kind, role, label, and link, and counts works with no link.
|
|
A reference link is resolved in this order: an explicit `url`, then `base_url` plus `path`, then the reference's `url`, then the digital object identifier (DOI), arXiv id, PubMed Central (PMC) id, and PubMed id.
|
|
`course build` merges the split course, and the course scripts of both repositories use its output to list lecture ids; the author's authoring pipelines read the same output as their course file.
|
|
|
|
## Migrating an older course
|
|
|
|
Every `migrate` subcommand takes `--dry-run`, which shows what would change and writes nothing.
|
|
The current schema version is `2.1`, and major versions 1 and 2 are readable, so a 1.0 or 2.0 course loads unchanged and migration is optional.
|
|
2.1 is additive: a course that validated under 2.0 validates under 2.1 with no new errors.
|
|
|
|
| Command | What it does |
|
|
|---------|-------------|
|
|
| `migrate split` | split one `course.yaml` into `references.yaml`, `lectures/`, and `objectives/`; keeps the original as `course.yaml.bak`; reassembles and compares before reporting success (exits 2 if the reassembly differs) |
|
|
| `migrate counters` | drop the trailing counter from every item id (`q-x-001` becomes `q-x`) across banks, records, seals, and the response store; colliding ids abort |
|
|
| `migrate order` | replace `order:` integers with ordered declarations; run `migrate split` first |
|
|
| `migrate references` | turn citations written into `note:` fields into real fields (journal, volume, pages, DOI); never overwrites a declared field |
|
|
| `migrate variants` | fill the stored `variant` column in `data/` from the assessment records |
|
|
| `migrate stems` | drop the ignored `version:` and `history:` fields |
|
|
| `migrate options` | rewrite option letters as names derived from option text; run with `--dry-run` first; seals keep their letters on purpose; exits 2 if some items need hand-naming |
|
|
| `migrate ids` | rewrite pre-2.0 `bank::item` ids as plain item ids in records, seals, and the response store |
|
|
| `migrate glossary --terms FILE --values FILE` | bring a 1.0 `terms.yaml` and `values.yaml` into `glossary.yaml`: ids gain `term-` and `val-`, `course` becomes `term`, `meaning` becomes `definition`, and `first` and `lectures` are dropped because 2.1 derives them; reports renamed ids, every dropped `first` or `lectures` the targets now contradict, and entries no target lists yet; leaves the old files in place and skips entries the course already declares, so a second run is a no-op |
|
|
|
|
Run `--dry-run` first, and commit before applying.
|
|
The hints that some `migrate` commands print afterwards name commands that do not exist (see the limits section).
|
|
|
|
## Command reference
|
|
|
|
coursebank has 21 top-level names and 50 runnable commands: 9 single commands and 12 groups holding 41 subcommands.
|
|
Every command accepts `--course PATH` (or `-C PATH`) and `--quiet` (or `-q`).
|
|
Identifiers are positional.
|
|
|
|
| Stage | Command | What it does |
|
|
|-------|---------|--------------|
|
|
| Project setup | `init --code C --title T --term TERM [--with-examples]` | create a course; refuses to overwrite `course.yaml` |
|
|
| | `schema` | write the nine JSON Schemas to `schema/` |
|
|
| | `validate` | check every file for problems that must be fixed; exits 2 on any |
|
|
| | `lint [flags]` | check item craft; see Authoring items |
|
|
| | `catalog [--coverage] [--topics]` | summarize the pool and objective coverage |
|
|
| | `course files`, `course build [--out PATH]`, `course where ID` | list the source files, print the merged course, find where an id is defined |
|
|
| | `references list`, `references export [--format F] [--out PATH]` | list or export the bibliography |
|
|
| Authoring | `bank new ID [--title T]`, `bank list` | create an empty bank, list banks |
|
|
| | `assessment new ID [--title T] [--kind K]`, `assessment list`, `assessment show ID` | create, list, or inspect a record |
|
|
| | `usage history [ITEM]`, `usage unused` | reuse history; approved items never used |
|
|
| Assembly | `assemble ID [flags]` | draw a new assessment from the pool |
|
|
| Sealing | `seal ID [--check] [--force] [--form A,B] [--no-content] [--out PATH]` | freeze what is printed; check for drift |
|
|
| Export | `export qti`, `export typst`, `export md`, `export practice`, `export site` | see Exporting |
|
|
| | `template list`, `template dump`, `template config` | manage Typst templates |
|
|
| Ingest | `ingest gradescope SOURCE...`, `ingest canvas FILE` | read a grading export |
|
|
| Analysis | `analyze items ID`, `analyze irt ID`, `analyze students ID` | analyze one assessment |
|
|
| | `calibrate` | pool every stored administration into `analysis/calibration.yaml` |
|
|
| | `report students ID`, `report cohort ID` | per-student and instructor reports |
|
|
| | `data` | list the stored administrations |
|
|
| Lectures | `lecture readings ID`, `lecture objectives ID`, `lecture page ID`, `lecture glossary`, `lecture coverage` | site partials; the whole lecture page; the course glossary page; reading coverage |
|
|
| Migration | `migrate split`, `counters`, `order`, `references`, `variants`, `stems`, `options`, `ids`, `glossary` | upgrade an older course; all take `--dry-run` |
|
|
|
|
coursebank has no `import`, `guide`, or `completions` subcommand.
|
|
The only help is `--help`, and the guide pages are rustdoc modules.
|
|
|
|
## Conventions that apply to every command
|
|
|
|
- Course location: `--course PATH` or `-C PATH` names the directory that holds `course.yaml` (default: the current directory). A command that loads the course fails with a usage error if there is no `course.yaml`, and the message says to run `init` or to use `--course`.
|
|
- Version: `--version` prints the build-time version string.
|
|
- Quiet: `--quiet` or `-q` suppresses some trailing hints and advisories.
|
|
- Exit codes: `0` is success, `1` is a failure (printed to standard error as `error: ...`), and `2` means the command ran and found problems. Exit 2 comes from `validate` (any problem), `lint` (any finding, unless `--no-fail`), `catalog --coverage` (gaps), `lecture coverage` (an assessed target without a reading), `assessment show` (problems in the record), `analyze items` (a non-empty revise queue), `report cohort` (a non-empty revise list), `seal --check` (drift), `export typst` (any template warning), `template dump` (a skipped file), `migrate split` (reassembly differs), and `migrate options` (items needing hand-naming). Code 2 is why a course repository can have a pre-commit hook.
|
|
- Unknown keys are errors: a misspelled field in a YAML file fails validation instead of being ignored.
|
|
- Dates: every date is written `YYYY-MM-DD`; anything else is an error.
|
|
- Count flags: options such as `--levels` and `--bonus` take `LEVEL=COUNT` with a level from 1 to 5; `--require` takes `OBJECTIVE=COUNT`.
|
|
- Assessment lookup: an assessment is found by file name (`assessments/ID.yaml`) or, failing that, by the `id:` inside any record. A form is found by its id, case-insensitively, and a record that declares no forms gets an implicit form `A`.
|
|
- No runtime environment variables: only the build-time `COURSEBANK_VERSION` exists. The pseudonymization salt is passed as a file, not an environment variable.
|
|
- Determinism: no command uses system entropy for the draw. The one exception is the site export, which uses the system random generator for passwords and initialization vectors.
|
|
|
|
## Using coursebank as a Rust library
|
|
|
|
The package builds two targets: the binary `coursebank` from `src/main.rs` and the library `coursebank` from `src/lib.rs`.
|
|
The crate forbids unsafe code and warns on undocumented public items.
|
|
The public modules are:
|
|
|
|
| Module | Contents |
|
|
|--------|----------|
|
|
| `model` | the data model and validation: `assessment`, `bank`, `calibration`, `catalog`, `course`, `history`, `item`, `layout`, `seal`, `taxonomy` |
|
|
| `authoring` | `lint`, `select` (the assembly draw), `jsonschema` |
|
|
| `data` | `canvas`, `decode`, `gradescope`, `intake`, `responses`, `store`, `store_parquet` |
|
|
| `analysis` | `calibrate`, `classical`, `diagnostic`, `irt`, `students` |
|
|
| `export` | `lecture`, `practice`, `qti`, `references`, `report`, `site`, `typst` |
|
|
| `util` | `citation`, `date`, `hash`, `markup`, `rng`, `yaml`, `zipfile` |
|
|
| `error`, `migrate`, `guide` | the error type, migrations, and the guide pages |
|
|
|
|
The crate root re-exports these submodules and the main types: `Catalog`, `CourseFile`, `SCHEMA_VERSION`, `Error`, `Result`, `History`, `Item`, `Layout`, and from the taxonomy `CognitiveProcess`, `ErrorType`, `Flag`, `Format`, `Level`, and `Status`.
|
|
`coursebank::VERSION` is the package version, not the nightly string that `--version` can show.
|
|
|
|
The commands call these library modules, so a program can do the same work: load a course, assemble an exam, and analyze stored responses.
|
|
The calls below come from the guide pages, where each is a doctest that is compiled without being run [CHECK: this block combines fragments from three guide pages and was not compiled as a whole; imports, error handling, and the construction of `blueprint` are left out, and return types such as `Result` may differ].
|
|
|
|
```rust
|
|
// Load a course and every bank, with calibration attached to each item.
|
|
let catalog = Catalog::load(Path::new("path/to/course"));
|
|
|
|
// Draw items. History comes from the assessment records.
|
|
let history = History::load(&catalog.layout.assessments());
|
|
let selection = select::select(&catalog, &blueprint, &history, Date::new(2026, 3, 24)?);
|
|
|
|
// Analyze a stored administration.
|
|
let store = Store::open("data");
|
|
let responses = store.read_assessment("exam-2");
|
|
let analysis = classical::analyze(&responses, &Thresholds::default(), None, None);
|
|
let queue = analysis.revise_queue();
|
|
```
|
|
|
|
The rustdoc guide modules, in order, are `coursebank::guide::setup`, `authoring`, `first_exam`, `typst_export`, `assignment`, and `recipes`.
|
|
The Rust examples in them are doctests, most of them marked `no_run`, so `pixi run tests` compiles them.
|
|
`pixi run doc` builds the API documentation and opens it, and `pixi run doc-build` builds it without opening a browser.
|
|
|
|
## Worked picture: biosc1000
|
|
|
|
biosc1000 is a biochemistry course (term 2026f) whose repository holds two things: `course/`, the coursebank data, and `website/`, a Quarto site.
|
|
|
|
### Shape and scale
|
|
|
|
- Course directory: `course.yaml` (a root fragment with identity, policy, units, and one stimulus), 17 lecture files, 82 objective files, and `references.yaml`.
|
|
- Merged course: `full-course.yaml` has 4 units, 17 lectures, 82 learning objectives, 399 learning targets, and 12 references, and is git-ignored.
|
|
- Banks: 13 banks with 339 items (311 approved, 16 in review, 12 draft), 336 of them single best answer and 3 multiple response.
|
|
- Assessment records: 15, made up of 13 homework sets and 2 exams.
|
|
- Statistics: no bank item carries observed statistics; the banks hold only the authored predictions in `design`.
|
|
The statistics are in `analysis/calibration.yaml`, which has 70 calibrated items, 34 from the first exam and 36 from the second, with 2PL parameters on 36 of them.
|
|
- Committed: four administration CSVs (an items file and an options file per exam) are committed.
|
|
|
|
### How the tool is called
|
|
|
|
The binary is a 16 MB file at `course/coursebank`, git-ignored and not on `PATH`, and it reports version `0.0.0`.
|
|
Scripts reach it through the variable `COURSEBANK`, whose default is `./coursebank`.
|
|
The repository has no wrapper of its own that calls it.
|
|
Instead, a set of shared course scripts, installed once per machine as commands, do the calling.
|
|
They read a per-course `course.env` (as data, never executed) and an `exams.tsv` manifest.
|
|
|
|
| Script | What it calls |
|
|
|--------|---------------|
|
|
| `course-merge` | `course build --out` to write the merged course |
|
|
| `lectures-export` | `course build` to list lectures, then `lecture objectives` and `lecture readings` for each, into the website's `_objectives.qmd` and `_readings.qmd` |
|
|
| `assessments-render` | `export site`, `export typst`, `export qti`, and `export practice` per assessment |
|
|
| `exams-seal` | `seal` for each id in `SEAL_IDS` |
|
|
| `exams-process` | `validate`, `seal --check`, `ingest`, `analyze items`, `report`, `calibrate`, `lint` |
|
|
|
|
Across all of these, the scripts use 15 distinct command forms: `course build`, `seal` (and `seal --check`), `export site`, `export typst`, `export qti`, `export practice`, `lecture objectives`, `lecture readings`, `validate`, `ingest` (Canvas and Gradescope), `analyze items` (and `--record`), `report cohort`, `report students`, `calibrate` (and `--apply`), and `lint` [CHECK: the figure of 15 comes from the source sheet; the count depends on whether the flag variants are counted separately].
|
|
|
|
### Homework to the site and Canvas, exams to paper
|
|
|
|
`assessments-render` chooses the formats from the kind of assessment.
|
|
Homework gets a site page (`export site`), Typst, and a QTI zip for Canvas.
|
|
Exams get Typst only, because they are on paper.
|
|
Recitations would get the site and practice exports.
|
|
The 13 homework sets each have a generated `_questions.qmd` and an encrypted `ID-solutions.json` under `website/assignments/`.
|
|
The partials carry a banner saying they are generated and must not be edited, and `lectures-export --check` fails if they are out of date.
|
|
The site exporter does not write the question level, so the script injects a level label from the record, with a comment to retire that once the exporter does it.
|
|
Page passwords are sticky: a rerun reuses the stored password, and a key rotates only when `OVERWRITE=true` is set.
|
|
The script also skips a page whose input hash (the record, all banks, `course.yaml`, and the binary) is unchanged.
|
|
|
|
### An exam through the loop
|
|
|
|
Each exam follows the order that the scripts encode.
|
|
|
|
1. Author the bank and the assessment record. The exams have two forms each (A and B, with `shuffle_items: false` and `shuffle_options: true`), 37 items each, and 75 minutes. The first exam's blueprint is 6, 18, 9, and 3 items at levels 1 to 4 plus one level-5 bonus item.
|
|
2. `exams-seal` writes `seals/ID.yaml` before printing. Homework on Canvas has no seal.
|
|
3. Print from the Typst files in `build/`.
|
|
4. Grade in Gradescope. The manifest `exams.tsv` lists, for each exam, one directory per form (`A=DIR B=DIR`) and the flags `--date` and `--format csv`; a directory means a Gradescope export and a single file means a Canvas Student Analysis CSV, detected from the path [CHECK: the biosc1000 sheet could not confirm that the Gradescope ingest was actually run; it infers it from the manifest and the stored CSVs].
|
|
5. `exams-process` runs per assessment: `validate` (skipped by default), `seal --check`, `ingest ... --dry-run` and then the real ingest, `analyze items` (look), a pause for "any question to drop or regrade?", and `analyze items --record`. It then compiles the cohort and per-student reports with Typst.
|
|
6. The scrub step: the first ingest keeps ids so that the reports can be routed. The script re-ingests with `--pseudonymize --salt-file` after the reports are compiled, so that what stays in `data/` has no identifiers. The script refuses a salt that lives inside the repository.
|
|
7. Calibration runs once, after the loop: `calibrate`, then (after a confirmation) `calibrate --apply`, then `validate`, and finally `lint`, which runs after calibration because five lint rules compare evidence with predictions.
|
|
|
|
Stored data is CSV: the manifest passes `--format csv`, and the data directory holds two CSV files, one per exam, with one row per examinee and item (66 by 37 and 65 by 37) and no Parquet file.
|
|
The first exam's seal records two dropped questions.
|
|
|
|
### Where the outputs go
|
|
|
|
The scripts write to the three destinations in [Where results are stored](#where-results-are-stored): `data/`, `analysis/`, and `reports/`.
|
|
The repository's README also states that student data, the salt, answer-key PDFs, and build output are never committed.
|
|
|
|
### The authoring pipelines around coursebank
|
|
|
|
The banks are written by a separate multiple-choice authoring pipeline that drafts a module bank (the module exam and the homework on the same lectures, planned together) and moves each item's status from `draft` to `approved`.
|
|
Its last stage runs `coursebank validate`, and when the binary is not installed in a sandbox, the pipeline falls back to its own lint and records that `coursebank validate` is still owed.
|
|
A separate lecture pipeline reads the `coursebank course build` output as its course file.
|
|
Those pipelines are not part of coursebank.
|
|
|
|
The record for one homework (A3.2) was written by hand, and the author's notes keep the equivalent `assemble` command, which also shows `--require` in use.
|
|
|
|
```sh
|
|
coursebank assemble A3.2 --title "A3.2 - Enzyme kinetics" --kind homework \
|
|
--date 2026-10-20 --platform canvas --levels 1=2,2=9,3=3,4=1 --bonus 5=1 \
|
|
--require lo-rate-regimes=4,lo-kinetic-parameters=5,lo-catalytic-efficiency=2,lo-kinetic-data=5 \
|
|
--lectures L3.2 --seed 3202 --forms 1
|
|
```
|
|
|
|
## Worked picture: biosc1540
|
|
|
|
biosc1540 is a computational biology course (term 2026f) with the same shape as biosc1000, built on older copies of the scripts that live inside the repository, and the differences show what else the tool handles.
|
|
|
|
### Shape and scale
|
|
|
|
- Course directory: `course.yaml` (a 42-line identity, policy, and units file), 14 lecture files, 70 objective files (69 objectives and 356 targets), and `references.yaml` with 64 entries.
|
|
- Banks: 11 banks with 284 items (238 approved, 45 retired, 1 draft), all single best answer.
|
|
- Assessment records: 13 (11 homework, 2 exams).
|
|
- Item design: every item has a `design` block, every option has `feedback_student`, 854 of the 1,138 options name a `misconception` and an `error_type`, and no bank item carries observed statistics.
|
|
- Statistics: `analysis/calibration.yaml` has 72 items, 36 from each exam, and 36 of them have 2PL IRT with priors.
|
|
|
|
### The scripts inside the repository
|
|
|
|
`scripts/` holds 10 helper scripts, of which five call coursebank:
|
|
|
|
- `merge-course.sh` runs `course build --out full-course.yaml`.
|
|
- `export-lectures.sh` reads lecture ids from the `course build` output and runs `lecture objectives` and `lecture readings` for each, with `--check` and `--prune` options.
|
|
- `seal-exams.sh` runs `seal` for each exam.
|
|
- `render-assessments.sh` runs the four exports, with the same format defaults as in biosc1000.
|
|
- `process-exams.sh` runs the whole post-exam loop from `exams.tsv`, and has the same scrub step and the same refusal of a salt inside the repository, and it hard-codes the binary as `./coursebank`.
|
|
|
|
Each script changes into `course/` first and calls `./coursebank` or `$COURSEBANK`.
|
|
`render-assessments.sh` includes the binary in its change hash, so replacing the binary makes every page re-render, while the stored passwords are reused.
|
|
|
|
### What is different here
|
|
|
|
- Migrated from the 1.0 layout: a commit titled "migrate to new coursebank" rewrote `course.yaml` and split lectures and objectives into their own files. A mix of schema versions remains: 5 banks and 6 assessments are at schema 1.0, and 6 banks and 7 assessments are at 2.0, and `course.yaml` still declares `"1.0"`. The schema text in the course mentions `coursebank migrate order`. The mix works because both major versions are readable.
|
|
- Multi-form sealed exams: the first exam declares three forms (A, B, and C, with `shuffle_items: false` and `shuffle_options: true`) and the second two (A and B) [CHECK: the manifest row for each exam lists two form directories, so confirm whether the first exam's third form was graded separately]. Both exams are sealed so that printed letters map back to option ids at ingest.
|
|
- Dropped items and bonus items: the second exam has four items marked `dropped: true` with `dropped_as: full_credit`, which the stored data carries in its `dropped` and `dropped_full_credit` columns. Both exams use bonus items (a level 3 and a level 5 bonus on the first, a level 5 bonus on the second), and several homework sets mark bonus items.
|
|
- Gradescope ingest stored as CSV: the manifest gives one directory per form and `--format csv`, so the two stores in `data/` are CSV files with no Parquet file. The analysis records say 27 examinees for each administration.
|
|
- Calibration with IRT: `calibrate` and `calibrate --apply` produced the 72-item calibration file.
|
|
- The salt outside the repository: the salt's default location is a folder outside the repo, the `COURSEBANK_SALT` variable overrides it, and the script refuses a path inside the repo.
|
|
- Reports: a cohort PDF and a per-student PDF (27 per exam) are produced with Typst, using the student flags `--ability --no-comparison --no-study --no-focus --no-strengths`. The reports are git-ignored.
|
|
- No custom templates: the repository has no `templates/` directory and no `typst.yaml`, so the built-in templates are used.
|
|
- Recitation practice is wired and unexercised: `render-assessments.sh` has the `export practice` path (worksheet and solutions, rendered to PDF by Quarto). No recitation record exists, so that path has not been run in this repository.
|
|
- Homework to the site: all 11 homework sets have a generated page with a stored password. The site keys are tracked in git although the repository's own notes say to never commit them.
|
|
|
|
## What the two repositories teach about adopting the tool
|
|
|
|
The two repositories are the only evidence of the tool in use, and they agree on several habits.
|
|
|
|
- Call the tool through shell scripts: the binary is a git-ignored file at `course/coursebank`, not on `PATH`, and the scripts change into the course directory, so `./coursebank` resolves without `-C`.
|
|
- The binary is not pinned: every seal and administration record in both repositories says the tool version is `0.0.0`, because no release has been cut. Nothing in either repository records how the binary was built or which commit it came from. Rebuild deliberately, and expect that an older seal or record was written by a different build than the one you have now. [CHECK: the sheets do not say how the author obtains or updates these binaries, or whether they were built from this repository's current head.]
|
|
- Keep the merged `full-course.yaml` out of git: it is derived by `course build` and rebuilt after editing fragments. Tools that need lectures read it, because `course.yaml` is only a root fragment.
|
|
- Assessment ids come from `id:`: file names are lowercase with hyphens, the ids are `A1.2` and `e1`, and every script, manifest row, and seal is keyed on the id.
|
|
- Generated partials say "do not edit": the site files that coursebank writes are regenerated, and a `--check` mode fails if they are stale.
|
|
- Keep student data out of git: student data, salts, reports, and answer-key PDFs are never committed. The salt lives outside every repository, and a scrub step removes identifiers from what stays on disk.
|
|
- Passwords for site solutions are sticky: a rerun reuses the stored password so a released password keeps working, and rotation is deliberate. Both repositories track their password files in git, against their own scripts' warning not to commit them on a public repository [CHECK: the brief mentioned one repository, but both source sheets report tracked password files; confirm].
|
|
|
|
Three decisions are yours to weigh, and the repositories give no clean answer to any of them.
|
|
The first is storage format. Both repositories pass `--format csv`, and the repositories do not record why. CSV opens in any editor, while Parquet is the default, is Snappy-compressed, and keeps column types.
|
|
The second is whether to commit site passwords. The render script refuses to touch an existing solutions bundle when the stored `_password.txt` is missing, so losing the file strands a released password; committing the file prevents that and allows anyone who can read the repository to open the solutions.
|
|
The third is when to calibrate. Running `calibrate` after each exam gives a running estimate, and running it once after the last exam gives a single pass, because it recomputes from every stored administration. The code labels any analysis with fewer than 100 examinees a small sample, and each administration in biosc1540 recorded 27, so weigh single-class statistics accordingly.
|
|
|
|
## Known limits and unfinished parts
|
|
|
|
Each item below states what the code does.
|
|
|
|
### Build and features
|
|
|
|
- The feature flags `parquet` and `site` appear in help text, the lean-build task, and the guides. `Cargo.toml` defines neither. Parquet storage and `export site` are always compiled in.
|
|
- `Cargo.lock` is git-ignored, although `pixi run install` uses `--locked`.
|
|
- The Rust version is declared as 1.85 in `Cargo.toml` and pinned to 1.96 in `pixi.toml`.
|
|
|
|
### Commands and flags
|
|
|
|
- `export qti --attempts` is accepted and ignored; attempts come from the assessment record.
|
|
- `export typst --form` takes one value or `all`, not a repeated flag.
|
|
- `export typst --variant student-report|cohort-report` is accepted by the parser; its behavior was not verified, so use `report` for those documents.
|
|
- `init --with-examples` writes only an empty bank, `banks/example.yaml`, and not an example assessment.
|
|
- `init` writes 9 schema files, not the 4 that `docs/guide/setup.md` shows.
|
|
- `policy.exam_conditions` and `policy.blueprints` are stored and validated, and no command reads them yet; `assemble` still takes its counts from the command line.
|
|
- `schema` prints a modeline path (`../.coursebank/schema`) that differs from where it writes the files (`schema/`).
|
|
- `--apply` exists only on `calibrate`, and it writes `analysis/calibration.yaml`, not a bank. The `src/main.rs` documentation says every command that modifies a bank prints what it would change and requires `--apply`; the `migrate` commands use `--dry-run` instead.
|
|
- There is no `import` subcommand, although `src/main.rs` and a `bank` comment mention a legacy importer.
|
|
- The guide pages use `analyze items --assessment exam-2` and similar forms. On the command line the id is positional, and `calibrate` has no assessment argument. The IRT model value is `two-pl`, not `2pl`.
|
|
- `lint --rules`, as shown in two guide pages, is `lint --list-rules`.
|
|
- `migrate counters` and `migrate ids` print follow-up hints that name `seal verify` and `analyze items --all`, neither of which exists. Use `seal --check` to verify.
|
|
|
|
### Analysis
|
|
|
|
- IRT in `calibrate` uses a single administration, so the pooled IRT parameters describe that one administration.
|
|
- `dif_flagged` is defined, and no code sets it. `high_rapid_guess` cannot fire because neither reader records response time. `mean_response_time_seconds` and `rapid_guess_rate` are never written to the calibration file.
|
|
- Student names are read from the export and do not appear in the stored schema, so a report built from stored data falls back to the student key.
|
|
|
|
### Markup and output
|
|
|
|
- `#sym.minus`, used in an authoring example, is not in the symbol table.
|
|
- Fenced code blocks in stems are not handled by the markup converter.
|
|
- The lecture and glossary pages leave `$...$` math and an equation's `tex` for Quarto to render, and a value's `say` is printed as written, so a `say` like `about 10^{59}` shows its caret and braces unless it is written in math markup.
|
|
- `conflicts` and `target_text_forbid` patterns use Rust's `regex` syntax, which has no look-around and no backreferences.
|
|
- Item figures (`assets`) go into the Typst payload only, and the bundled Typst templates do not print them. The QTI, practice, and site exports do not emit assets.
|
|
- The bundled Typst templates import `@preview` packages, which the Typst compiler resolves; whether they are cached or downloaded at compile time was not checked.
|
|
|
|
### Guides and older text
|
|
|
|
- The guide pages describe an item-level `calibration:` block written by `calibrate`, a `version:` field on items and placements, letter option ids, and `learning_objectives` on items. In the 2.0 code the statistics live in `analysis/calibration.yaml`, a `calibration:` block in a bank is a validation error, `version:` is ignored, items use `learning_targets`, and option ids are names with letters still accepted.
|
|
- `docs/guide/setup.md` says `cognitive_process` has eleven values (the code has 19) and tells authors to set `order` on objectives (which is deprecated and derived from `teaches`).
|
|
- A `docs/AUTHORING.md` that a code comment points to does not exist; the file is `docs/guide/authoring.md`.
|
|
|
|
### Release files
|
|
|
|
- No release tags exist, only the rolling `nightly` tag. The package version is `0.0.0` in `Cargo.toml`, `pixi.toml`, and `.bumpversion.toml`, and `CHANGELOG.md` has only an `[Unreleased]` section.
|
|
- `RELEASING.md` names workflows `ci.yml`, `nightly.yml`, `release.yml`, and `docs.yml`. The repository has `.gitea/workflows/pipeline.yml` and `sync-readme.yml`.
|
|
- `pipeline.yml`, `scripts/check-tag-version.sh`, and `scripts/dist-attach.sh` call `scripts/version.sh`, which does not exist.
|
|
- The `.bumpversion.toml` parse pattern requires the first number to start with 1 to 9, so `0.0.0` does not appear to match [CHECK: not run; confirm that `pixi run bump` works from `0.0.0`].
|
|
- The continuous integration and release files were read only shallowly for this README.
|
|
|
|
## Versioning and releases
|
|
|
|
Releases use calendar versioning in `YY.MM.PATCH` form.
|
|
The first release of August 2026 is `26.8.0`; a second that month is `26.8.1`; the first in September is `26.9.0`.
|
|
Neither the year nor the month is zero-padded, because the version also has to parse as Semantic Versioning (SemVer) and SemVer forbids leading zeros.
|
|
The number carries no compatibility promise, so treat any release as one that can change behavior and read [`CHANGELOG.md`](CHANGELOG.md) before upgrading a live course.
|
|
|
|
`pixi run bump` cuts the next version and tags it; pushing that tag publishes a release, and every push to `main` refreshes a rolling `nightly` prerelease.
|
|
`pixi run bump-show` previews the next version without changing anything.
|
|
The workflows live in `.gitea/workflows/`; [RELEASING.md](RELEASING.md) explains the release flow and what to configure on the Gitea side.
|
|
The pipeline workflow runs `pixi run check` first, on pushes to `main`, tag pushes, pull requests, a daily schedule, and manual dispatch.
|
|
The nightly job sets `COURSEBANK_VERSION` from `scripts/nightly-version.sh` (the last release tag plus the commit count and short hash, or `0.0.0` plus those when there is no tag), builds the binary, packages it as `coursebank-nightly-linux-x64.tar.gz`, and replaces the `nightly` prerelease.
|
|
The release job checks that the tag matches the crate version and publishes a Gitea release with the packaged binary and its checksum.
|
|
The documentation (rustdoc) is also published for the `release` and `nightly` channels.
|
|
|
|
The release flow has not been used yet, and parts of it are unfinished; [Known limits and unfinished parts](#known-limits-and-unfinished-parts) lists each one under Release files.
|
|
A build from source or the nightly archive is the way to get a binary until the first tag [CHECK: the sheets confirm that a `nightly` tag exists but not what archive it holds or whether the job has run successfully].
|
|
|
|
## Documentation
|
|
|
|
- `docs/guide/setup.md`: building a course directory, the references and readings YAML, editor schemas, `validate` versus `lint`, exit codes, and reading a course from Rust.
|
|
- `docs/guide/authoring.md`: `bank new`, the bank header, the smallest valid item, status gating, the markup, designed distractors, partial credit, the `design` block, and `lint`.
|
|
- `docs/guide/first_exam.md`: one exam end to end, from `assemble` to `calibrate`, including after-the-fact fixes.
|
|
- `docs/guide/assignment.md`: publishing a homework to a Quarto site with `export site`, password handling, and rotation.
|
|
- `docs/guide/recipes.md`: short answers on assembly flags, reuse, exports, ingest, analysis, reports, and housekeeping.
|
|
- `docs/TYPST.md`: the Typst export, markers and slots, `typst.yaml`, `reveal`, the lookup order, and the guarantees.
|
|
- These six files are compiled into the API documentation as `coursebank::guide` modules. They have known drift from the code, listed under [Known limits and unfinished parts](#known-limits-and-unfinished-parts), so trust this README and the code where they differ.
|
|
- [`CHANGELOG.md`](CHANGELOG.md), [`RELEASING.md`](RELEASING.md), [`LICENSE.md`](LICENSE.md), [`NOTICE.md`](NOTICE.md), and `THIRD-PARTY-LICENSES.txt`.
|
|
|
|
The source lives at `https://git.scient.ing/education/coursebank`.
|
|
This README is mirrored to a GitHub repository with a notice that the project's home is the Gitea address.
|
|
|
|
## License
|
|
|
|
coursebank is open source under the [Apache License 2.0](LICENSE.md).
|
|
Anyone, companies included, can use, modify, and redistribute it under that license's terms.
|
|
Attribution notices are in [`NOTICE.md`](NOTICE.md).
|
|
|
|
Bundled dependencies keep their own licenses, reproduced in `THIRD-PARTY-LICENSES.txt` in each release.
|
|
|
|
## Contributing
|
|
|
|
Open an issue first if you want to contribute.
|
|
Contributed code will need a signed contributor license agreement before it can be merged.
|
|
That agreement is not in place yet, so outside patches cannot be merged for now.
|