feat: robust docs
This commit is contained in:
@@ -1,3 +1,5 @@
|
||||
preview
|
||||
|
||||
# pixi environments
|
||||
.pixi/*
|
||||
!.pixi/config.toml
|
||||
|
||||
@@ -35,3 +35,7 @@ parquet = { version = "55", optional = true }
|
||||
[profile.release]
|
||||
opt-level = 3
|
||||
lto = "thin"
|
||||
|
||||
[package.metadata.docs.rs]
|
||||
all-features = true
|
||||
rustdoc-args = ["--cfg", "docsrs"]
|
||||
|
||||
+127
@@ -0,0 +1,127 @@
|
||||
# Typst export
|
||||
|
||||
The tool does not write Typst documents.
|
||||
It loads a Typst file you own, finds the markers in it, and injects data.
|
||||
Layout is yours; the payload is the tool's.
|
||||
|
||||
```console
|
||||
$ coursebank template dump # get the built-in templates as files
|
||||
$ coursebank template list # see which template each document uses
|
||||
$ coursebank template config # write templates/typst.yaml
|
||||
$ coursebank export typst exam-2 # render paper, key, and answer sheet
|
||||
```
|
||||
|
||||
## Markers
|
||||
|
||||
Injection points are Typst line comments, so a template is a valid `.typ` file that compiles on its own:
|
||||
|
||||
```typst
|
||||
// coursebank:begin questions
|
||||
#render-question((number: 1, stem: [Sample.], options: ()))
|
||||
// coursebank:end questions
|
||||
```
|
||||
|
||||
Everything *between* the markers is replaced.
|
||||
The marker lines survive.
|
||||
Two consequences:
|
||||
|
||||
- The bundled templates ship with sample data inside their regions, so `typst watch templates/exam.typ` works before you have exported anything.
|
||||
Restyle against the sample, then export.
|
||||
- **An exported document is itself a valid template.** Exporting into a file you have since restyled replaces the questions and leaves your edits alone.
|
||||
This is the difference between a generator you can use twice and one you copy out of once.
|
||||
|
||||
A bare `// coursebank:questions` also works.
|
||||
It is rewritten into a region on output, so the second export behaves like every one after it.
|
||||
|
||||
Malformed markers are reported all at once, with line numbers, and the export stops: an unknown slot name, an unclosed region, a stray `end`, a nested region, or the same slot claimed twice.
|
||||
|
||||
## Slots
|
||||
|
||||
| Slot | Injected |
|
||||
|:--|:--|
|
||||
| `meta` | `#let cb-meta = (...)`: course, assessment, form, totals, objectives |
|
||||
| `questions` | one `#render-question((...))` call per printed item |
|
||||
| `data` | `#let cb-data = (...)`: the metadata and the questions together |
|
||||
|
||||
`questions` unrolls the loop using the record's own numbering.
|
||||
`data` hands you the array and gets out of the way; the bundled key and answer sheet use it because a table suits a loop better than a sequence of calls.
|
||||
Use either, both, or neither.
|
||||
Only slots your template actually contains are built, so nothing costs anything until it is asked for.
|
||||
|
||||
Each question arrives as a **single positional dictionary**, not named arguments, so turning a field on or off in the config never changes your function's signature.
|
||||
Read optional fields with `q.at("level", default: none)`.
|
||||
|
||||
## Configuration
|
||||
|
||||
`templates/typst.yaml`, three layers, each overriding the last: built-in defaults for the variant, then `defaults:`, then `variants:`.
|
||||
|
||||
```yaml
|
||||
defaults:
|
||||
question_fn: render-question # the function the `questions` slot calls
|
||||
content: content # `content` -> [...] | `str` -> "..." for eval()
|
||||
letters: upper # upper | lower | numeric | roman | nothing
|
||||
extra:
|
||||
accent: "#017ab9"
|
||||
font: Roboto
|
||||
|
||||
variants:
|
||||
key:
|
||||
reveal: everything
|
||||
extra:
|
||||
show-solutions: true
|
||||
```
|
||||
|
||||
Keys are snake_case, matching every other coursebank YAML file.
|
||||
Keys *under* `extra` are yours and reach Typst verbatim, so they conventionally use hyphens.
|
||||
|
||||
`coursebank template config --resolved exam` prints what a variant actually ends up with, which is the quickest way to find out which layer won.
|
||||
|
||||
### `extra` is the escape hatch
|
||||
|
||||
Anything under `extra` is carried through untouched and arrives as `extra` in the payload.
|
||||
Tier colours, a font stack, a `show-solutions` flag, a watermark, column counts.
|
||||
Put it there and read it in the template.
|
||||
Nothing about appearance needs to be added to this crate.
|
||||
|
||||
### `reveal` is not cosmetic
|
||||
|
||||
`reveal` controls whether the payload contains the answer **at all**.
|
||||
The exam variant uses `nothing`, and that means an option dictionary on the paper has no `correct` field, not `correct: false`.
|
||||
A template cannot leak a field it was never given, and that stays true after someone edits the template without reading this page.
|
||||
|
||||
Do not set `reveal: key` on the exam variant to build a solutions copy.
|
||||
Export the `key` variant.
|
||||
The failure mode of the other approach is one forgotten `if` away, and it is discovered by the whole room at once.
|
||||
|
||||
## Template lookup
|
||||
|
||||
1. `--template <path>`, or `template:` in the render config
|
||||
2. `templates/<assessment-id>-<variant>.typ`, a one-off layout for one exam
|
||||
3. `templates/<variant>.typ`, the course's own default
|
||||
4. the template compiled into the binary
|
||||
|
||||
`coursebank template list` prints this chain with the resolved entry marked, plus the slots each template declares.
|
||||
`coursebank template dump` writes step 4 into step 3.
|
||||
|
||||
## JSON
|
||||
|
||||
`coursebank export typst exam-2 --json` also writes the payload as JSON, for a template that reads `json("exam-2-A.json")` instead of taking an injected region.
|
||||
Key spellings are identical between the two paths (`level-name`, not `level_name`), so a template can move between them without edits.
|
||||
JSON has no content type, so set `content: str` and `eval(q.stem, mode: "markup")` if you use this path.
|
||||
|
||||
## What is still guaranteed
|
||||
|
||||
**The key matches its paper.** Option order comes from the form's recorded seed via `select::option_order`, never from anything stored, and the paper, key, and answer sheet are built from one payload.
|
||||
Every export of form B agrees with every other.
|
||||
|
||||
**Question numbers are the recorded ones.**
|
||||
Not positions on the page.
|
||||
The recorded number is the join key to every grading export and response row; renumbering after a drop breaks that join silently, and the symptom is item statistics attributed to the wrong question.
|
||||
`number-from-record: false` exists but you almost certainly want it left alone.
|
||||
|
||||
## Advisory warnings
|
||||
|
||||
`export typst` exits `2` and prints warnings, without refusing to write, when:
|
||||
|
||||
- a stem or option has unbalanced `[` `]`, which would otherwise surface as a Typst parse error somewhere downstream of the item that caused it, with no way for the compiler to name the question
|
||||
- a template declares no markers at all, so nothing was injected
|
||||
@@ -0,0 +1,221 @@
|
||||
# Authoring items
|
||||
|
||||
An item is a question plus two things a question does not normally carry: what you predicted about it before anyone answered, and what happened when they did.
|
||||
Keeping those next to each other is what turns a pile of questions into an instrument you can improve, because every administration produces a prediction you can check.
|
||||
|
||||
## A bank
|
||||
|
||||
```console
|
||||
$ coursebank bank new sequence-analysis --title "Sequence analysis"
|
||||
wrote banks/sequence-analysis.yaml
|
||||
```
|
||||
|
||||
A bank is a topic grouping, not a unit of reuse. Items are drawn across banks by blueprint, so split banks by whatever makes them easy to edit.
|
||||
One per unit is a reasonable default.
|
||||
|
||||
The header declares scope and defaults:
|
||||
|
||||
```yaml
|
||||
bank:
|
||||
id: sequence-analysis
|
||||
title: Sequence analysis
|
||||
scope:
|
||||
units: [u1]
|
||||
lectures: [l09, l10, l11]
|
||||
|
||||
defaults:
|
||||
author: Alex Maldonado
|
||||
options_per_item: 5
|
||||
topics: [alignment]
|
||||
```
|
||||
|
||||
`defaults` fills in fields you would otherwise repeat on every item.
|
||||
|
||||
## The smallest item that validates
|
||||
|
||||
```yaml
|
||||
items:
|
||||
- id: q-align-recall-001
|
||||
version: 1
|
||||
status: approved
|
||||
level: 1
|
||||
cognitive_process: recall
|
||||
format: single_best_answer
|
||||
title: Needleman-Wunsch vs Smith-Waterman
|
||||
stem: |
|
||||
Which alignment algorithm guarantees an optimal *local* alignment between two sequences?
|
||||
options:
|
||||
- id: A
|
||||
text: Needleman#sym.minus Wunsch
|
||||
correct: false
|
||||
- id: B
|
||||
text: Smith#sym.minus Waterman
|
||||
correct: true
|
||||
learning_objectives: [lo-align-algorithm]
|
||||
sources:
|
||||
- lecture: l09
|
||||
slides: [12, 13]
|
||||
```
|
||||
|
||||
Ids are never reused and never renumbered.
|
||||
The id is the join key that ties an item to every assessment it has appeared on and every response row ever recorded for it, so `q-align-recall-001` stays that even after the stem is rewritten twice.
|
||||
|
||||
`status` gates assembly.
|
||||
Only `approved` items can be drawn onto a graded assessment, and approval requires the item to be fully specified: a cognitive process, an objective, a source, and a key.
|
||||
Draft items are visible to `lint` and invisible to `assemble`.
|
||||
|
||||
`level` and `cognitive_process` are checked against each other.
|
||||
`level: 1` with `cognitive_process: evaluate` is an error, not a warning, because one of the two is wrong and the tool cannot tell which.
|
||||
|
||||
## Markup
|
||||
|
||||
Stems are written in a small markup that is a subset of Typst with a few Markdown conveniences, because chemistry and biology need subscripts, arrows, and Greek letters, and typing HTML entities into YAML by hand is miserable.
|
||||
|
||||
```yaml
|
||||
stem: >
|
||||
A reaction proceeds at 37#sym.degree C with #sym.delta G = #sym.minus 12
|
||||
kJ/mol. Rate increases *linearly* with `[S]` below K_m.
|
||||
```
|
||||
|
||||
`#sym.arrow.r`, `#sym.alpha`, `#sym.gt.eq`, and the rest of the table render as arrows and Greek in all three outputs.
|
||||
Emphasis uses `*bold*` and `_italic_`, and backticks give monospace.
|
||||
The same source becomes HTML for Canvas, Typst for print, and plain text for CSV, so you write it once.
|
||||
|
||||
## Distractors that earn their place
|
||||
|
||||
The optional fields on an option are what separate a designed distractor from filler:
|
||||
|
||||
```yaml
|
||||
- id: A
|
||||
text: Needleman#sym.minus Wunsch
|
||||
correct: false
|
||||
misconception: |
|
||||
They remember that both are dynamic programming and pick the more familiar name without distinguishing global from local
|
||||
error_type: recall_confusion
|
||||
explanation: |
|
||||
Needleman#sym.minus Wunsch is the global algorithm; it aligns the full length of both sequences.
|
||||
feedback_student: |
|
||||
Needleman#sym.minus Wunsch is the global algorithm.
|
||||
Both use dynamic programming, so the distinction to hold onto is what happens at the matrix boundaries and where the traceback starts.
|
||||
```
|
||||
|
||||
When a third of the cohort picks that option, you know what they were thinking, and the student report can tell each of them specifically rather than saying "incorrect, the answer was B."
|
||||
`error_type` is one of thirteen categories, which is what lets cohort analysis say the class is losing points to dropped steps rather than to terminology.
|
||||
|
||||
`explanation` is for you. `feedback_student` is released to students afterwards and is the text a report shows someone who chose that option. `misconception` is used for both when neither of the others is written, so a partly-authored item degrades gracefully instead of producing a blank.
|
||||
|
||||
### Partial credit
|
||||
|
||||
A wrong option that is defensible can earn credit, but only with the argument written down:
|
||||
|
||||
```yaml
|
||||
- id: C
|
||||
text: Nothing can be said without replicates
|
||||
correct: false
|
||||
credit: 0.5
|
||||
defensible: true
|
||||
defense: >
|
||||
A descriptive question about a single pair of libraries admits this
|
||||
reading, so it earns half credit rather than zero.
|
||||
```
|
||||
|
||||
`defense` is required whenever credit goes to a wrong option.
|
||||
That is deliberate.
|
||||
Partial credit decided in the moment and never recorded becomes a decision you cannot reconstruct next term, and then you relitigate it with the next student who asks.
|
||||
|
||||
The course policy's `partial_credit_floor_level` applies here.
|
||||
Credit awarded below that level is flagged, on the theory that a reasonable wrong answer to a recall question means the question is unclear.
|
||||
|
||||
## Predictions
|
||||
|
||||
The `design` block is what you think before anyone sits the exam:
|
||||
|
||||
```yaml
|
||||
design:
|
||||
expected_difficulty: 0.72
|
||||
expected_discrimination: moderate
|
||||
expected_time_seconds: 55
|
||||
rationale: |
|
||||
Recall of a named distinction taught in one slide.
|
||||
Most of the cohort should get it; the ones who miss it are confusing the two algorithms rather than failing to recall either.
|
||||
```
|
||||
|
||||
`expected_time_seconds` summed over a form is how you check that an exam fits the period, which is the most common way a well-written exam goes wrong.
|
||||
|
||||
The other two are checkable predictions.
|
||||
After the exam, `lint` compares them against what happened and reports the misses.
|
||||
An item you expected to be easy that two thirds of the class missed is either mis-taught or mis-written, and either way you want to be told.
|
||||
|
||||
## Statistics come back
|
||||
|
||||
You do not write the `calibration` block.
|
||||
`coursebank calibrate` does, after `ingest` and `analyze`:
|
||||
|
||||
```yaml
|
||||
calibration:
|
||||
administrations: [exam-2-2026s, exam-2-2025s]
|
||||
updated: 2026-04-02
|
||||
fingerprint: 8f3a2c...
|
||||
n_examinees: 47
|
||||
p_value: 0.68
|
||||
point_biserial: 0.31
|
||||
flags: []
|
||||
```
|
||||
|
||||
Calibration is cumulative rather than per administration.
|
||||
Raw per-response data lives in the Parquet tables under `data/`, which are much better at holding it, and the item's YAML keeps the rolled-up estimate plus a list of which administrations went into it.
|
||||
Bank files stay readable in a pull request while statistics accumulate across terms.
|
||||
Twenty-four students tells you very little; ninety-six across four terms tells you something.
|
||||
|
||||
The `fingerprint` is why this is safe.
|
||||
It covers only what a student saw: the stem, the option text, and the key.
|
||||
Retag an item's metadata and the pooled statistics stay valid.
|
||||
Reword the stem and the fingerprint changes, the numbers are marked stale, and the linter says so rather than letting you trust a p-value from a question that no longer exists.
|
||||
|
||||
## Lint before you commit
|
||||
|
||||
```console
|
||||
$ coursebank lint
|
||||
banks/sequence-analysis.yaml
|
||||
q-align-gap-002 cue-uneven-length the key is 1.8x the average distractor length (94 vs 52)
|
||||
q-dock-analyze-002 clarity-stem-length stem runs 84 words
|
||||
|
||||
2 finding(s)
|
||||
```
|
||||
|
||||
`coursebank lint --rules` lists every rule with its code.
|
||||
Silence one you disagree with; the codes exist so that disagreeing is a configuration change rather than a reason to stop running the linter.
|
||||
|
||||
## Checking a bank from Rust
|
||||
|
||||
```rust,no_run
|
||||
use coursebank::bank::BankFile;
|
||||
use coursebank::Status;
|
||||
|
||||
# fn main() -> coursebank::Result<()> {
|
||||
let bank = BankFile::load(std::path::Path::new("banks/sequence-analysis.yaml"))?;
|
||||
|
||||
let approved = bank
|
||||
.items
|
||||
.iter()
|
||||
.filter(|item| item.status == Status::Approved)
|
||||
.count();
|
||||
|
||||
println!("{approved} of {} items are assemblable", bank.items.len());
|
||||
|
||||
for item in &bank.items {
|
||||
if let Some(calibration) = &item.calibration {
|
||||
if !item.calibration_is_current() {
|
||||
println!("{}: statistics predate the current wording", item.id);
|
||||
} else if let Some(p) = calibration.p_value {
|
||||
println!("{}: p = {p:.2}", item.id);
|
||||
}
|
||||
}
|
||||
}
|
||||
# Ok(())
|
||||
# }
|
||||
```
|
||||
|
||||
## Next
|
||||
|
||||
[`first_exam`](crate::guide::first_exam) draws a form from this bank and follows it through grading.
|
||||
@@ -0,0 +1,287 @@
|
||||
# One exam, end to end
|
||||
|
||||
This follows a single exam from blueprint to student report, using the sequence analysis and docking banks.
|
||||
It assumes a course directory with approved items in it; if you do not have one, [`setup`](crate::guide::setup) and [`authoring`](crate::guide::authoring) build one.
|
||||
|
||||
The loop:
|
||||
|
||||
```text
|
||||
author items ──▶ validate ──▶ lint ──▶ assemble ──▶ export ───┐
|
||||
▲ │
|
||||
│ administer
|
||||
│ │
|
||||
calibrate ◀── analyze ◀── ingest ◀───────────────────────────┘
|
||||
│
|
||||
└──▶ report (students and cohort)
|
||||
```
|
||||
|
||||
The arrow back from `calibrate` to authoring is the point of the design.
|
||||
Statistics land on the item, so they are there the next time you consider using it.
|
||||
|
||||
## Draw a form
|
||||
|
||||
Describe the exam you want by level, not by item:
|
||||
|
||||
```console
|
||||
$ coursebank assemble exam-2 \
|
||||
--title "Exam 2 — Sequence analysis and docking" \
|
||||
--kind exam --date 2026-03-24 --platform paper \
|
||||
--levels 1=2,2=1,3=2,4=1 --bonus 5=1 \
|
||||
--require lo-align-scoring=2,lo-dock-scoring=1 \
|
||||
--max-per-bank 4 --cooldown 180 --forms 2 --seed 20260324
|
||||
```
|
||||
|
||||
`--levels 1=2,2=1,3=2,4=1` asks for six scored items across four cognitive levels.
|
||||
`--bonus 5=1` adds one level-5 item outside the scored total, which is where level-5 work belongs on a timed multiple-choice paper.
|
||||
|
||||
`--require` sets floors per objective, so an exam cannot accidentally measure the scoring objective with a single question.
|
||||
`--cooldown 180` avoids items used in the last six months, computed by scanning assessment records rather than by consulting a separate ledger.
|
||||
There is no ledger file, because a ledger duplicates what the records must already get right and then drifts from it.
|
||||
|
||||
Use `--dry-run` first.
|
||||
It prints the draw without writing anything, and a blueprint that cannot be satisfied tells you which constraint failed rather than silently returning fewer items.
|
||||
|
||||
What lands in `assessments/exam-2.yaml` is a record of what happened, not a plan:
|
||||
|
||||
```yaml
|
||||
items:
|
||||
- number: 1
|
||||
item: sequence-analysis::q-align-recall-001
|
||||
version: 1
|
||||
points: 1.5
|
||||
key: [B]
|
||||
level: 1
|
||||
learning_objectives: [lo-align-algorithm]
|
||||
```
|
||||
|
||||
Level and objectives are denormalized onto the placement so the record reads standalone in five years, whatever the bank says by then.
|
||||
|
||||
## Check it against the blueprint
|
||||
|
||||
```console
|
||||
$ coursebank assessment show exam-2
|
||||
Exam 2 — Sequence analysis and docking 2026-03-24 paper 50 min
|
||||
6 scored items, 9.0 points; 1 bonus item, 1.5 points
|
||||
levels: 1×2 2×1 3×2 4×1
|
||||
estimated time: 41 minutes of 50 allowed
|
||||
blueprint: satisfied
|
||||
```
|
||||
|
||||
The time estimate sums each item's `expected_time_seconds`, falling back to a level-based guess for items with no `design` block.
|
||||
An exam that does not fit the period is the most common way a well-written exam goes wrong, and it is invisible until you are standing in the room.
|
||||
|
||||
## Export
|
||||
|
||||
Two forms with shuffled options, plus a key and a bubble sheet for each:
|
||||
|
||||
```console
|
||||
$ coursebank export typst exam-2 --form all
|
||||
wrote build/exam-2-A.typ (from built-in)
|
||||
wrote build/exam-2-A-key.typ (from built-in)
|
||||
wrote build/exam-2-A-answer-sheet.typ (from built-in)
|
||||
wrote build/exam-2-B.typ (from built-in)
|
||||
...
|
||||
```
|
||||
|
||||
Option order comes from each form's recorded seed, never from anything stored, so form B's key is generated from the same permutation that produced form B's paper.
|
||||
A key that disagrees with its paper is discovered by twenty-five students at once.
|
||||
|
||||
The `(from built-in)` note means no template override was found.
|
||||
`coursebank template dump` writes the defaults into `templates/` so you can restyle them; see [`typst_export`](crate::guide::typst_export).
|
||||
|
||||
Compile with `pixi run -e docs typst compile build/exam-2-A.typ`.
|
||||
|
||||
For a Canvas quiz instead:
|
||||
|
||||
```console
|
||||
$ coursebank export qti exam-2 --form A
|
||||
wrote build/exam-2-A.zip
|
||||
Import in Canvas: Settings -> Import Course Content -> QTI .zip file
|
||||
```
|
||||
|
||||
## Ingest the grading export
|
||||
|
||||
After the exam, read the grader's output into the response store:
|
||||
|
||||
```console
|
||||
$ coursebank ingest gradescope grading/exam-2/ \
|
||||
--assessment exam-2 --form A --pseudonymize --salt-file ~/.coursebank-salt
|
||||
read 24 students × 7 items = 168 rows
|
||||
wrote data/exam-2-2026s.parquet
|
||||
```
|
||||
|
||||
`--pseudonymize` replaces student identifiers with HMAC pseudonyms keyed by a salt you keep outside the repository.
|
||||
Without the salt, hashed ids can be reversed by brute force over a class roster; with the salt committed next to them, so can they.
|
||||
The generated `.gitignore` excludes `*.salt` for that reason.
|
||||
|
||||
Use `--dry-run` on a new export format.
|
||||
Gradescope's per-question CSVs vary, and parsing 168 rows wrong is easier to see in a report than in a Parquet file.
|
||||
|
||||
## Analyze
|
||||
|
||||
```console
|
||||
$ coursebank analyze items --assessment exam-2
|
||||
# p rpb flags
|
||||
1 0.88 0.21
|
||||
2 0.71 0.34
|
||||
3 0.46 0.09 low-discrimination
|
||||
4 0.63 0.41
|
||||
5 0.54 0.18 ambiguous
|
||||
6 0.29 -0.12 negative-discrimination
|
||||
7 0.21 0.15 bonus
|
||||
|
||||
reliability: KR-20 = 0.61 (24 examinees, 6 scored items)
|
||||
Caution: with 6 items, reliability is limited by test length as much as by
|
||||
item quality.
|
||||
|
||||
3 item(s) need revision
|
||||
```
|
||||
|
||||
Read the corrected point-biserial first.
|
||||
It correlates each item against the total of the *other* items, which answers the question you actually care about: did the students who knew the material get this right? A negative value almost always means the key is wrong, so check that before rewriting anything.
|
||||
|
||||
Item 6 above is the one to look at tonight.
|
||||
Item 3's low discrimination is expected if it is an anchor item and worth investigating if it sits at level 3 or higher.
|
||||
|
||||
Every statistic computed from a class of twenty-four is reported with the caveat it deserves rather than three decimal places of false precision.
|
||||
|
||||
For a fuller picture:
|
||||
|
||||
```console
|
||||
$ coursebank analyze irt --assessment exam-2 --model 2pl
|
||||
$ coursebank analyze students --assessment exam-2
|
||||
```
|
||||
|
||||
## Report
|
||||
|
||||
```console
|
||||
$ coursebank report students --assessment exam-2
|
||||
wrote 24 report(s) to reports/exam-2/
|
||||
$ coursebank report cohort --assessment exam-2
|
||||
wrote reports/exam-2-cohort.md
|
||||
```
|
||||
|
||||
These are two documents with different content, not different tones.
|
||||
The student report answers "what should I do next?" and deliberately omits correct answers, other students' data, and any numeric rank.
|
||||
Where a student chose a designed distractor, it names the misconception that distractor was built to catch and points at the lecture and slides.
|
||||
The cohort report answers "what should I fix?" and holds the item statistics.
|
||||
|
||||
## Write the statistics back
|
||||
|
||||
```console
|
||||
$ coursebank calibrate --assessment exam-2
|
||||
q-align-recall-001 p 0.71 -> 0.68 rpb 0.29 -> 0.31 n 23 -> 47
|
||||
q-dock-analyze-002 NEW p 0.29 rpb -0.12 n 24 flag: negative-discrimination
|
||||
...
|
||||
7 item(s) would change. Re-run with --apply to write.
|
||||
```
|
||||
|
||||
Every command that modifies a bank prints what it would change and requires `--apply`.
|
||||
These are reviewed artifacts in a git repository, and a silent rewrite is not something you want to discover in a diff later.
|
||||
|
||||
```console
|
||||
$ coursebank calibrate --assessment exam-2 --apply
|
||||
```
|
||||
|
||||
Now the pooled statistics are on the items, and next term's `assemble` sees them.
|
||||
|
||||
## When grading reveals a problem
|
||||
|
||||
Two fields get added to the assessment record by hand, after the fact, and both stay there so that next term's analysis knows the exam was scored the way it was actually scored.
|
||||
|
||||
An option that turned out to be defensible earns partial credit:
|
||||
|
||||
```yaml
|
||||
- number: 5
|
||||
item: structure-and-expression::q-rnaseq-explain-004
|
||||
points: 1.5
|
||||
key: [B]
|
||||
# Decided during grading: option C ("nothing can be said without replicates")
|
||||
# is a defensible reading of a descriptive question, so it earns half credit.
|
||||
credit_overrides:
|
||||
C: 0.5
|
||||
```
|
||||
|
||||
Recording it here rather than editing scores by hand means item analysis sees the same numbers the students did.
|
||||
|
||||
An item that was broken gets dropped:
|
||||
|
||||
```yaml
|
||||
- number: 6
|
||||
item: structure-and-expression::q-dock-analyze-002
|
||||
dropped: true
|
||||
```
|
||||
|
||||
Dropped items leave the scored matrix and are not printed on a re-export, but the placement stays in the record, because the fact that the question was asked is part of what happened.
|
||||
|
||||
Both of these make `analyze items` flag the item as ambiguous, which is the correct outcome.
|
||||
The fix is to rewrite the stem so the narrower question is unambiguous, not to relitigate the partial credit every term.
|
||||
|
||||
## Doing this from Rust
|
||||
|
||||
The CLI is a thin wrapper.
|
||||
Assembling a form programmatically:
|
||||
|
||||
```rust,no_run
|
||||
use std::collections::BTreeMap;
|
||||
use std::path::Path;
|
||||
|
||||
use coursebank::assessment::{Blueprint, History};
|
||||
use coursebank::date::Date;
|
||||
use coursebank::{select, Catalog, Level};
|
||||
|
||||
# fn main() -> coursebank::Result<()> {
|
||||
let catalog = Catalog::load(Path::new("."))?;
|
||||
|
||||
let mut level_counts = BTreeMap::new();
|
||||
level_counts.insert(Level::Remember, 2);
|
||||
level_counts.insert(Level::Understand, 1);
|
||||
level_counts.insert(Level::Apply, 2);
|
||||
|
||||
let blueprint = Blueprint {
|
||||
level_counts,
|
||||
max_per_bank: Some(4),
|
||||
cooldown_days: Some(180),
|
||||
seed: Some(20260324),
|
||||
..Blueprint::default()
|
||||
};
|
||||
|
||||
// Usage history is derived by scanning the assessment records, so cooldowns are
|
||||
// measured against what was actually given rather than a separate ledger.
|
||||
let history = History::load(&catalog.layout.assessments())?;
|
||||
let selection = select::select(&catalog, &blueprint, &history, Date::new(2026, 3, 24)?)?;
|
||||
|
||||
for uid in &selection.scored {
|
||||
println!("scored: {uid}");
|
||||
}
|
||||
for note in &selection.notes {
|
||||
// Quotas filled by relaxing a constraint say so here.
|
||||
println!("note: {note}");
|
||||
}
|
||||
# Ok(())
|
||||
# }
|
||||
```
|
||||
|
||||
Reading responses back and running item analysis:
|
||||
|
||||
```rust,no_run
|
||||
use coursebank::classical::{self, Thresholds};
|
||||
use coursebank::store::Store;
|
||||
|
||||
# fn main() -> coursebank::Result<()> {
|
||||
let store = Store::open("data")?;
|
||||
// `read` takes an administration id; `read_assessment` gathers every
|
||||
// administration of one assessment across terms.
|
||||
let responses = store.read_assessment("exam-2")?;
|
||||
|
||||
let analysis = classical::analyze(&responses, &Thresholds::default(), None, None);
|
||||
|
||||
for item in analysis.revise_queue() {
|
||||
println!(
|
||||
"item {}: p = {:.2}, {:?}",
|
||||
item.number, item.p_value, item.flags
|
||||
);
|
||||
}
|
||||
# Ok(())
|
||||
# }
|
||||
```
|
||||
@@ -0,0 +1,119 @@
|
||||
# Recipes
|
||||
|
||||
Short answers, for when you know the shape of the tool and want the invocation.
|
||||
|
||||
## Assembly
|
||||
|
||||
**Draw only from material I have taught.**
|
||||
`--lectures l09,l10,l11`.
|
||||
Combine with `--topics` and `--banks` to narrow further; the filters intersect.
|
||||
|
||||
**See the draw before committing to it.**
|
||||
`--dry-run`.
|
||||
Prints the selection and any notes about constraints that had to bend, and writes nothing.
|
||||
|
||||
**Reproduce a draw exactly.**
|
||||
`--seed N`.
|
||||
The same seed against the same pool gives the same items in the same order.
|
||||
Recorded in the assessment file, so a draw stays reproducible after the fact.
|
||||
|
||||
**Two forms that differ only in option order.**
|
||||
`--forms 2`.
|
||||
Each form gets its own seed; item order is shared unless the form sets `shuffle_items`.
|
||||
|
||||
**A blueprint I cannot satisfy.**
|
||||
The error names the level, how many items were asked for, and how many were available after filtering.
|
||||
Usually the fix is a shorter cooldown or a wider lecture range, not more items.
|
||||
|
||||
## Reuse
|
||||
|
||||
```console
|
||||
$ coursebank usage history q-align-recall-001
|
||||
$ coursebank usage unused
|
||||
```
|
||||
|
||||
`unused` lists approved items never placed on an assessment, which is the queue of work you already did and forgot about.
|
||||
|
||||
## Exports
|
||||
|
||||
**A printable exam.**
|
||||
`coursebank export typst exam-2 --form all`.
|
||||
Add `--variant key` to write only the key.
|
||||
|
||||
**A Canvas quiz.**
|
||||
`coursebank export qti exam-2 --form A`.
|
||||
Add `--no-feedback` to leave per-option feedback out of the package.
|
||||
|
||||
**A Markdown copy for a colleague to read.**
|
||||
`coursebank export md exam-2`.
|
||||
Add `--with-key` for the answers and rationales.
|
||||
|
||||
**Restyle the printed output.**
|
||||
`coursebank template dump`, then edit `templates/exam.typ`.
|
||||
See [`typst_export`](crate::guide::typst_export).
|
||||
|
||||
## Ingest
|
||||
|
||||
**Gradescope.**
|
||||
`coursebank ingest gradescope grading/exam-2/ --assessment exam-2`.
|
||||
Point it at the directory holding the per-question CSVs.
|
||||
|
||||
**Canvas.**
|
||||
`coursebank ingest canvas export.csv --assessment exam-2`.
|
||||
The Student Analysis export, not the gradebook.
|
||||
|
||||
**Keep student identities out of the repository.**
|
||||
`--pseudonymize --salt-file ~/.coursebank-salt`.
|
||||
Keep the salt outside the repository; the point of the salt is that hashed ids cannot be brute-forced over a class roster, which fails if the salt sits next to them.
|
||||
|
||||
**Check a parse before writing.** `--dry-run`.
|
||||
|
||||
## Analysis
|
||||
|
||||
| Question | Command |
|
||||
|:--|:--|
|
||||
| Which items misbehaved? | `analyze items --assessment exam-2` |
|
||||
| How hard is each item, on a common scale? | `analyze irt --assessment exam-2 --model 2pl` |
|
||||
| Which students are struggling, and with what? | `analyze students --assessment exam-2` |
|
||||
| What is in the store? | `data` |
|
||||
|
||||
Pool across terms by passing the assessment id rather than one administration id.
|
||||
Twenty-four students supports very little; ninety-six across four terms supports something.
|
||||
|
||||
## Reports
|
||||
|
||||
```console
|
||||
$ coursebank report students --assessment exam-2
|
||||
$ coursebank report cohort --assessment exam-2
|
||||
```
|
||||
|
||||
The student report omits correct answers, other students' data, and any rank.
|
||||
Hand it out without a second pass.
|
||||
|
||||
## After grading
|
||||
|
||||
**An option turned out to be defensible.**
|
||||
Add `credit_overrides: {C: 0.5}` to the placement in the assessment record.
|
||||
Do not edit scores by hand, or item analysis sees different numbers than the students did.
|
||||
|
||||
**An item was broken.**
|
||||
Add `dropped: true` to the placement.
|
||||
It leaves the scored matrix and is not printed on re-export, but the record of having asked it stays.
|
||||
|
||||
**Statistics onto the items.**
|
||||
`coursebank calibrate --assessment exam-2`, read the diff, then `--apply`.
|
||||
|
||||
## Housekeeping
|
||||
|
||||
**Editor validation stopped working.**
|
||||
`coursebank schema` rewrites the JSON Schemas.
|
||||
They ship with the binary, so an upgrade can leave them stale.
|
||||
|
||||
**A pre-commit hook.**
|
||||
`coursebank validate && coursebank lint`.
|
||||
Exit code `2` means findings, `1` means the command failed, so a hook can treat them differently.
|
||||
|
||||
**Build without Parquet.**
|
||||
`pixi run build-lean`.
|
||||
The response store falls back to CSV.
|
||||
Useful if you want a binary with a shorter dependency list; the tradeoff is slower reads on large stores.
|
||||
@@ -0,0 +1,188 @@
|
||||
# Setting up a course
|
||||
|
||||
A course is a directory in a git repository.
|
||||
Nothing lives in a database, and the tool holds no state of its own, so a course you set up in 2026 opens in 2031 with whatever version of coursebank you have then.
|
||||
|
||||
## Make the directory
|
||||
|
||||
```console
|
||||
$ coursebank init --code "BIOSC 1540" --title "Computational Biology" --term 2026s
|
||||
wrote course.yaml
|
||||
wrote 4 JSON Schema files
|
||||
```
|
||||
|
||||
That gives you:
|
||||
|
||||
| Path | Holds | Written by |
|
||||
|:--|:--|:--|
|
||||
| `course.yaml` | identity, grading policy, objectives, lectures | you |
|
||||
| `banks/` | items, with design intent and pooled statistics | you, then `calibrate` |
|
||||
| `assessments/` | what was given, to whom, when | `assemble`, then you |
|
||||
| `data/` | one row per student per item | `ingest` |
|
||||
| `reports/` | generated Markdown and HTML | `report` |
|
||||
| `build/` | exports: QTI packages, `.typ` files, PDFs | `export` |
|
||||
| `schema/` | JSON Schemas for editor validation | `init`, `schema` |
|
||||
|
||||
`build/` and `reports/` are in the generated `.gitignore`.
|
||||
The other four are the repository's content and belong in review.
|
||||
|
||||
Add `--with-examples` if you want a filled-in bank to read rather than an empty directory to stare at.
|
||||
|
||||
## Point your editor at the schemas
|
||||
|
||||
The schemas are the difference between authoring items and looking up field names.
|
||||
With them wired in, your editor completes `cognitive_process` from the eleven legal values and underlines a typo in `learning_objectives` as you type.
|
||||
|
||||
Put the modeline at the top of each file:
|
||||
|
||||
```yaml
|
||||
# yaml-language-server: $schema=../schema/bank.schema.json
|
||||
```
|
||||
|
||||
`coursebank schema` reprints the paths and the exact line to paste.
|
||||
Rerun it after upgrading, since the schemas ship with the binary.
|
||||
|
||||
## Fill in `course.yaml`
|
||||
|
||||
Four registries live here, and everything else references them by id.
|
||||
|
||||
### Policy
|
||||
|
||||
Conventions stated once instead of per assessment:
|
||||
|
||||
```yaml
|
||||
policy:
|
||||
points_per_item: 1.5
|
||||
options_per_item: 5
|
||||
bonus_levels: [5]
|
||||
allow_partial_credit: true
|
||||
partial_credit_floor_level: 3
|
||||
mastery_threshold: 0.75
|
||||
min_items_for_mastery: 3
|
||||
```
|
||||
|
||||
`partial_credit_floor_level: 3` is the one to think about.
|
||||
Below Apply, a defensible wrong answer usually means the item is unclear rather than that the student partly understood something.
|
||||
Setting a floor makes that a rule you decided once, so it stops being an argument you have every term with a student at your desk.
|
||||
|
||||
### Units and lectures
|
||||
|
||||
Units are the coarse grouping.
|
||||
Lectures carry a date and belong to a unit:
|
||||
|
||||
```yaml
|
||||
units:
|
||||
- id: u1
|
||||
title: Sequence analysis
|
||||
description: Alignment, scoring models, and database search.
|
||||
|
||||
lectures:
|
||||
l09:
|
||||
title: Pairwise alignment
|
||||
date: 2026-02-10
|
||||
unit: u1
|
||||
readings:
|
||||
- "Durbin et al., ch. 2"
|
||||
```
|
||||
|
||||
The dates are what let a student report say which lecture to review, and what lets `assemble --lectures l09,l10` draw only from material you have taught.
|
||||
|
||||
### Learning objectives
|
||||
|
||||
The load-bearing registry.
|
||||
An objective's wording lives in exactly one place, so rewording it updates every report that quotes it:
|
||||
|
||||
```yaml
|
||||
learning_objectives:
|
||||
lo-align-algorithm:
|
||||
text: Trace the dynamic programming recurrence for a global or local alignment and explain what each term contributes.
|
||||
unit: u1
|
||||
lectures: [l09]
|
||||
level_ceiling: 4
|
||||
tags: [algorithms]
|
||||
|
||||
lo-align-scoring:
|
||||
text: Predict how changing a substitution matrix or gap penalty changes the resulting alignment.
|
||||
unit: u1
|
||||
lectures: [l10]
|
||||
prerequisites: [lo-align-algorithm]
|
||||
level_ceiling: 4
|
||||
```
|
||||
|
||||
Write the text in the second person and start with a verb, because reports quote it verbatim to students.
|
||||
|
||||
Three fields do work later that is easy to miss now.
|
||||
`prerequisites` is walked backwards by student reports to suggest where to start reviewing, so a student who missed the scoring objective gets pointed at the algorithm first.
|
||||
`level_ceiling` is the highest level you intend to assess the objective at; placing an item above it is a warning, which means either the item overreaches or the ceiling needs raising, and both are useful to be asked about.
|
||||
`assessed: false` marks an objective you teach but measure some other way, such as by project rubric, which stops coverage reporting from flagging it as a gap on every run.
|
||||
|
||||
Objective ids are join keys.
|
||||
Renaming one orphans every item and every stored response that referenced it, so pick names you can live with.
|
||||
|
||||
### Stimuli
|
||||
|
||||
A shared passage, table, or figure that several items ask about:
|
||||
|
||||
````yaml
|
||||
stimuli:
|
||||
s-dock-poses:
|
||||
body: |
|
||||
A docking run produces five poses of the same ligand. Scores are in
|
||||
kcal/mol; RMSD is measured against the crystallographic pose.
|
||||
|
||||
```
|
||||
Pose Score RMSD (Å) Cluster size
|
||||
1 -9.8 6.2 3
|
||||
2 -9.4 1.1 28
|
||||
```
|
||||
caption: Docking output for a single ligand against one receptor.
|
||||
````
|
||||
|
||||
Items reference it with `stimulus: s-dock-poses`.
|
||||
Declaring it here rather than pasting it into four items means a correction to the table happens once.
|
||||
|
||||
## Check it
|
||||
|
||||
```console
|
||||
$ coursebank validate
|
||||
course.yaml: ok
|
||||
banks: 0 files, 0 items
|
||||
assessments: 0 records
|
||||
```
|
||||
|
||||
`validate` enforces what must be true: every reference resolves, ids are unique, keys are present, credit is in range.
|
||||
It reports everything wrong in one pass instead of one problem per run, because fixing one typo per invocation is not a workflow.
|
||||
|
||||
`lint` is separate and advises on what is usually a mistake: an option that gives away the answer by being longer than the others, a stem with no task in it, a level that disagrees with the cognitive process.
|
||||
Every rule has a code you can silence.
|
||||
The split matters because a linter that blocks a commit for a style opinion gets disabled, and then you lose the validator with it.
|
||||
|
||||
Exit codes are meaningful.
|
||||
`0` means success, `1` means the command failed, and `2` means validation or linting found something.
|
||||
|
||||
## Where to go next
|
||||
|
||||
[`authoring`](crate::guide::authoring) writes the first bank.
|
||||
[`first_exam`](crate::guide::first_exam) takes an exam from blueprint to student report.
|
||||
|
||||
## Reading a course from Rust
|
||||
|
||||
The CLI is one caller.
|
||||
[`Catalog::load`](crate::catalog::Catalog::load) reads a whole course directory and indexes every item by global id:
|
||||
|
||||
```rust,no_run
|
||||
use std::path::Path;
|
||||
|
||||
use coursebank::Catalog;
|
||||
|
||||
# fn main() -> coursebank::Result<()> {
|
||||
let catalog = Catalog::load(Path::new("path/to/course"))?;
|
||||
|
||||
println!("{} items across {} banks", catalog.entries.len(), catalog.banks.len());
|
||||
|
||||
// Global ids are `bank::item`. Bare ids resolve when unambiguous.
|
||||
let entry = catalog.require("sequence-analysis::q-align-recall-001")?;
|
||||
println!("{}", entry.item.display_title());
|
||||
# Ok(())
|
||||
# }
|
||||
```
|
||||
@@ -22,15 +22,20 @@ format = "cargo fmt"
|
||||
lint = { cmd = "cargo clippy --all-targets -- -D warnings", depends-on = [
|
||||
"format",
|
||||
] }
|
||||
doc = "cargo doc --no-deps --open"
|
||||
|
||||
check-docs = "cargo test --test docs"
|
||||
doc = { cmd = "cargo doc --no-deps --open", depends-on = ["check-docs"] }
|
||||
doc-build = { cmd = "cargo doc --no-deps", depends-on = ["check-docs"] }
|
||||
doctests = "cargo test --doc"
|
||||
|
||||
clean = "cargo clean"
|
||||
install = "cargo install --path . --locked"
|
||||
check = { depends-on = ["format", "lint", "tests"] }
|
||||
check = { depends-on = ["format", "lint", "tests", "check-docs", "doc-build"] }
|
||||
build-lean = "cargo build --release --no-default-features"
|
||||
|
||||
[tasks.cb]
|
||||
cmd = "cargo run --release --quiet --"
|
||||
description = "Run the CLI, e.g. `pixi run cb validate --course examples/course`"
|
||||
description = "Run the CLI, e.g., `pixi run cb validate --course examples/course`"
|
||||
|
||||
[feature.dev.dependencies]
|
||||
rust-analyzer = "*"
|
||||
|
||||
+1
-1
@@ -17,7 +17,7 @@
|
||||
//! contain those routinely.
|
||||
//!
|
||||
//! [`students`] turns item statistics into per-objective standing, and is careful
|
||||
//! about what three questions can honestly support: it classifies on the observed
|
||||
//! about what three questions can honestly support — it classifies on the observed
|
||||
//! rate and reports confidence from the Wilson interval separately.
|
||||
//!
|
||||
//! [`calibrate`] is the arrow back to authoring, and the reason the system
|
||||
|
||||
@@ -738,7 +738,7 @@ mod tests {
|
||||
|
||||
fn calibration(p: f64, rpb: Option<f64>, n: usize, fingerprint: &str) -> Calibration {
|
||||
Calibration {
|
||||
administrations: vec!["C/2026S/e1".into()],
|
||||
administrations: vec!["C/2026s/e1".into()],
|
||||
updated: None,
|
||||
fingerprint: Some(fingerprint.to_string()),
|
||||
n_examinees: Some(n),
|
||||
@@ -845,7 +845,7 @@ mod tests {
|
||||
}],
|
||||
unmatched: vec!["question 9".into()],
|
||||
warnings: vec!["only 24 examinees".into()],
|
||||
administrations: vec!["C/2026S/e1".into()],
|
||||
administrations: vec!["C/2026s/e1".into()],
|
||||
};
|
||||
let text = plan.render();
|
||||
assert!(text.contains("bank::q-a-001 (new)"));
|
||||
|
||||
@@ -1004,7 +1004,7 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn objective_counts_credit_every_tagged_item() {
|
||||
let rows = vec![
|
||||
let rows = [
|
||||
make("s1", 1, 1.0, &["lo-a", "lo-b"], Some(Level::Remember)),
|
||||
make("s1", 2, 0.0, &["lo-a"], Some(Level::Apply)),
|
||||
];
|
||||
@@ -1020,7 +1020,7 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn level_rates_ignore_untagged_items() {
|
||||
let rows = vec![
|
||||
let rows = [
|
||||
make("s1", 1, 1.0, &[], Some(Level::Remember)),
|
||||
make("s1", 2, 0.0, &[], None),
|
||||
];
|
||||
|
||||
@@ -209,7 +209,7 @@ fn course_identity_schema() -> Value {
|
||||
"properties": {
|
||||
"code": text("Course code, e.g. BIOSC 1540."),
|
||||
"title": text("Course title."),
|
||||
"term": text("Term, e.g. 2026S."),
|
||||
"term": text("Term, e.g., 2026s."),
|
||||
"institution": { "type": "string" },
|
||||
"instructors": string_array("Instructor names."),
|
||||
"slug": {
|
||||
|
||||
+1
-1
@@ -570,7 +570,7 @@ mod tests {
|
||||
|
||||
let ctx = Context {
|
||||
course: "BIOSC1540".into(),
|
||||
term: "2026S".into(),
|
||||
term: "2026s".into(),
|
||||
assessment_id: "quiz-1".into(),
|
||||
date: None,
|
||||
form: None,
|
||||
|
||||
@@ -823,7 +823,7 @@ mod tests {
|
||||
|
||||
let ctx = Context {
|
||||
course: "BIOSC1540".into(),
|
||||
term: "2026S".into(),
|
||||
term: "2026s".into(),
|
||||
assessment_id: "exam-4".into(),
|
||||
date: None,
|
||||
form: None,
|
||||
|
||||
@@ -40,7 +40,7 @@ pub struct Response {
|
||||
pub administration_id: String,
|
||||
/// The course code.
|
||||
pub course: String,
|
||||
/// The term, e.g. `2026S`.
|
||||
/// The term, e.g. `2026s`.
|
||||
pub term: String,
|
||||
/// The assessment id.
|
||||
pub assessment_id: String,
|
||||
@@ -456,7 +456,7 @@ impl ResponseSet {
|
||||
///
|
||||
/// # Returns
|
||||
///
|
||||
/// A stable identifier such as `BIOSC1540/2026S/exam-4`.
|
||||
/// A stable identifier such as `BIOSC1540/2026s/exam-4`.
|
||||
pub fn administration_id(course: &str, term: &str, assessment: &str) -> String {
|
||||
format!("{course}/{term}/{assessment}")
|
||||
}
|
||||
@@ -703,9 +703,9 @@ mod tests {
|
||||
|
||||
fn row(student: &str, number: u32, credit: f64) -> Response {
|
||||
Response {
|
||||
administration_id: "C/2026S/e1".into(),
|
||||
administration_id: "C/2026s/e1".into(),
|
||||
course: "C".into(),
|
||||
term: "2026S".into(),
|
||||
term: "2026s".into(),
|
||||
assessment_id: "e1".into(),
|
||||
date: None,
|
||||
form: None,
|
||||
|
||||
+4
-4
@@ -560,9 +560,9 @@ mod tests {
|
||||
|
||||
fn row(student: &str, number: u32) -> Response {
|
||||
Response {
|
||||
administration_id: "BIOSC1540/2026S/exam-4".into(),
|
||||
administration_id: "BIOSC1540/2026s/exam-4".into(),
|
||||
course: "BIOSC1540".into(),
|
||||
term: "2026S".into(),
|
||||
term: "2026s".into(),
|
||||
assessment_id: "exam-4".into(),
|
||||
date: None,
|
||||
form: None,
|
||||
@@ -591,7 +591,7 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn sanitizes_administration_ids() {
|
||||
assert_eq!(sanitize("BIOSC1540/2026S/exam-4"), "biosc1540_2026s_exam-4");
|
||||
assert_eq!(sanitize("BIOSC1540/2026s/exam-4"), "biosc1540_2026s_exam-4");
|
||||
assert_eq!(sanitize("///"), "responses");
|
||||
// Runs of separators collapse rather than stacking underscores.
|
||||
assert_eq!(sanitize("a // b"), "a_b");
|
||||
@@ -610,7 +610,7 @@ mod tests {
|
||||
assert_eq!(written.len(), 1);
|
||||
assert!(written[0].exists());
|
||||
|
||||
let back = store.read("BIOSC1540/2026S/exam-4").unwrap();
|
||||
let back = store.read("BIOSC1540/2026s/exam-4").unwrap();
|
||||
assert_eq!(back.rows.len(), 2);
|
||||
assert_eq!(back.rows[0].item_ref.as_deref(), Some("bank::q-x-001"));
|
||||
assert_eq!(back.rows[0].selected, vec!["C".to_string()]);
|
||||
|
||||
@@ -307,9 +307,9 @@ mod tests {
|
||||
|
||||
fn flat(student: &str, number: u32) -> FlatResponse {
|
||||
FlatResponse {
|
||||
administration_id: "C/2026S/e1".into(),
|
||||
administration_id: "C/2026s/e1".into(),
|
||||
course: "C".into(),
|
||||
term: "2026S".into(),
|
||||
term: "2026s".into(),
|
||||
assessment_id: "e1".into(),
|
||||
date: "2026-04-01".into(),
|
||||
form: String::new(),
|
||||
|
||||
@@ -434,7 +434,7 @@ impl RenderConfig {
|
||||
///
|
||||
/// # Errors
|
||||
///
|
||||
/// Returns [`Error::Other`](crate::error::Error::Other) if serialization
|
||||
/// Returns [`Error::Other`] if serialization
|
||||
/// fails.
|
||||
pub fn to_yaml(&self) -> Result<String> {
|
||||
serde_yaml_ng::to_string(self).map_err(Error::other)
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
|
||||
// coursebank:begin data
|
||||
#let cb-data = (
|
||||
course: (code: "COURSE 101", title: "Sample Course", term: "2026S"),
|
||||
course: (code: "COURSE 101", title: "Sample Course", term: "2026s"),
|
||||
assessment: (id: "sample", title: "Sample assessment", date: "2026-01-01"),
|
||||
form: (id: "A", count: 1),
|
||||
totals: (questions: 1, scored: 1, bonus: 0, points: 1.5, bonus-points: 0.0),
|
||||
@@ -49,8 +49,7 @@
|
||||
#grid(
|
||||
columns: (auto, 1fr, auto, 1fr),
|
||||
gutter: 0.6em,
|
||||
[*Name*], box(width: 100%, repeat[.]),
|
||||
[*Student ID*], box(width: 100%, repeat[.]),
|
||||
[*Name*], box(width: 100%, repeat[.]), [*Student ID*], box(width: 100%, repeat[.]),
|
||||
)
|
||||
|
||||
#v(1em)
|
||||
|
||||
@@ -8,7 +8,7 @@
|
||||
// typst watch templates/exam.typ restyle it against the sample data
|
||||
//
|
||||
// The regions ship with sample values so that last command works before any
|
||||
// export has happened.
|
||||
// export has happened. Two markers are in play:
|
||||
//
|
||||
// // coursebank:begin meta a dictionary of course and form metadata
|
||||
// // coursebank:end meta
|
||||
@@ -18,15 +18,19 @@
|
||||
// An exported document keeps its markers, so exporting into a file you have since
|
||||
// restyled replaces the questions and leaves the styling alone.
|
||||
//
|
||||
// There is no `correct` field on an option, because the render config for the
|
||||
// paper withholds it. That is deliberate. Do not switch `reveal` to `key`
|
||||
// here in order to build a solutions copy: export the `key` variant instead,
|
||||
// or the day you forget an `if` is the day the class gets the answers.
|
||||
// Note what is *not* in the payload for this variant: there is no `correct` field
|
||||
// on an option, because the render config for the paper withholds it. That is
|
||||
// deliberate. Do not switch `reveal` to `key` here in order to build a solutions
|
||||
// copy — export the `key` variant instead, or the day you forget an `if` is the
|
||||
// day the class gets the answers.
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
// Metadata
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
// coursebank:begin meta
|
||||
#let cb-meta = (
|
||||
course: (code: "COURSE 101", title: "Sample Course", term: "2026S"),
|
||||
course: (code: "COURSE 101", title: "Sample Course", term: "2026s"),
|
||||
assessment: (
|
||||
id: "sample",
|
||||
title: "Sample assessment",
|
||||
@@ -40,6 +44,9 @@
|
||||
)
|
||||
// coursebank:end meta
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
// Settings
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
// Anything under `extra` in templates/typst.yaml arrives here untouched, which is
|
||||
// how a course changes the look without editing this file at all.
|
||||
@@ -71,6 +78,9 @@
|
||||
#set text(font: body-font, size: body-size, lang: "en")
|
||||
#set par(justify: false, leading: 0.65em)
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
// Helpers
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
// Markup arrives as content when the render config says `content: content`, and as
|
||||
// a string when it says `content: str`. Accepting both means switching that
|
||||
@@ -110,6 +120,10 @@
|
||||
}
|
||||
}
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
// The renderer
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
//
|
||||
// One question, one function. Rename it if you like and set `question-fn` in
|
||||
// templates/typst.yaml to match. It takes a single dictionary so that turning a
|
||||
// field on or off in the config never changes this signature.
|
||||
@@ -152,10 +166,14 @@
|
||||
if page-per-item { pagebreak(weak: true) }
|
||||
}
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
// The page
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
#align(center)[
|
||||
#text(size: 1.4em, weight: "bold", fill: accent)[#cb-meta.assessment.title]\
|
||||
#text(size: 0.95em)[
|
||||
#cb-meta.course.code — #cb-meta.course.title · #cb-meta.course.term
|
||||
#cb-meta.course.code — #cb-meta.course.title · #cb-meta.assessment.term
|
||||
]\
|
||||
#text(size: 0.9em)[
|
||||
#cb-meta.assessment.at("date", default: "")
|
||||
|
||||
@@ -15,7 +15,7 @@
|
||||
|
||||
// coursebank:begin data
|
||||
#let cb-data = (
|
||||
course: (code: "COURSE 101", title: "Sample Course", term: "2026S"),
|
||||
course: (code: "COURSE 101", title: "Sample Course", term: "2026s"),
|
||||
assessment: (id: "sample", title: "Sample assessment", date: "2026-01-01"),
|
||||
form: (id: "A", count: 1),
|
||||
totals: (questions: 1, scored: 1, bonus: 0, points: 1.5, bonus-points: 0.0),
|
||||
@@ -63,7 +63,8 @@
|
||||
columns: (auto, auto, auto, auto, 1fr),
|
||||
align: (right, center, center, right, left),
|
||||
table.header([*\#*], [*Key*], [*Level*], [*Pts*], [*Objectives*]),
|
||||
..cb-data.questions
|
||||
..cb-data
|
||||
.questions
|
||||
.map(q => (
|
||||
[#q.number],
|
||||
[*#q.at("key", default: ()).join("")*],
|
||||
@@ -71,7 +72,7 @@
|
||||
[#fmt-points(q.at("points", default: 0))],
|
||||
[#objectives-of(q)],
|
||||
))
|
||||
.flatten()
|
||||
.flatten(),
|
||||
)
|
||||
|
||||
// ── Partial credit ──
|
||||
@@ -88,7 +89,7 @@
|
||||
.pairs()
|
||||
.map(pair => pair.at(0) + " = " + str(int(calc.round(pair.at(1) * 100))) + "%")
|
||||
[Question #q.number: #parts.join(", ")]
|
||||
})
|
||||
}),
|
||||
)
|
||||
}
|
||||
}
|
||||
@@ -137,7 +138,8 @@
|
||||
grid(
|
||||
columns: (1.2em, 1.4em, 1fr),
|
||||
gutter: 0.3em,
|
||||
[#marker], [#(opt.letter + ".")],
|
||||
[#marker],
|
||||
[#(opt.letter + ".")],
|
||||
[
|
||||
#markup(opt.text)
|
||||
#{
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
//! Long-form documentation: setup, authoring, and worked tutorials.
|
||||
//!
|
||||
//! Everything in this module is prose. The modules below hold no code, no types,
|
||||
//! and no runtime cost; each one exists so that a markdown file under `docs/guide/`
|
||||
//! gets a page in these docs, a slot in the sidebar, and a stable URL that
|
||||
//! [intra-doc links](https://doc.rust-lang.org/rustdoc/write-documentation/linking-to-items-by-name.html)
|
||||
//! elsewhere in the crate can point at.
|
||||
//!
|
||||
//! ## Read in this order
|
||||
//!
|
||||
//! The sidebar sorts alphabetically, which is not reading order. This is:
|
||||
//!
|
||||
//! 1. [`setup`] builds a course directory from nothing and explains what each file
|
||||
//! is for.
|
||||
//! 2. [`authoring`] writes items, with the design fields that make an item worth
|
||||
//! reusing.
|
||||
//! 3. [`first_exam`] runs one exam end to end: assemble, export, administer,
|
||||
//! ingest, analyze, report.
|
||||
//! 4. [`typst_export`] covers printed output, template markers, and render config.
|
||||
//! 5. [`recipes`] holds short answers to specific questions, for when you already
|
||||
//! know the shape of the tool.
|
||||
//!
|
||||
//! ## Why the tutorials are in here rather than a wiki
|
||||
//!
|
||||
//! Rust examples in these pages are doctests. `cargo test --doc` compiles every
|
||||
//! one of them against the crate as it currently is, so renaming
|
||||
//! [`Catalog::require`](crate::catalog::Catalog::require) breaks the documentation
|
||||
//! build rather than leaving a page that lies. Most examples carry `no_run`,
|
||||
//! because they want a course directory on disk that a test runner does not have.
|
||||
//! `no_run` still type-checks, which is where the value is.
|
||||
//!
|
||||
//! Shell transcripts get a `console` fence and YAML gets a `yaml` fence, so
|
||||
//! rustdoc leaves them alone. A fence with no language is Rust as far as rustdoc is
|
||||
//! concerned, and a `course.yaml` snippet in a bare fence fails the doc build with
|
||||
//! a parse error pointing at the markdown. `pixi run check-docs` catches that
|
||||
//! before the compiler has to.
|
||||
|
||||
/// Building a course directory, and what each file in it is for.
|
||||
#[doc = include_str!("../docs/guide/setup.md")]
|
||||
pub mod setup {}
|
||||
|
||||
/// Writing items that are worth keeping.
|
||||
#[doc = include_str!("../docs/guide/authoring.md")]
|
||||
pub mod authoring {}
|
||||
|
||||
/// One exam from blueprint to student report.
|
||||
#[doc = include_str!("../docs/guide/first_exam.md")]
|
||||
pub mod first_exam {}
|
||||
|
||||
/// Printed exams: templates, markers, and render configuration.
|
||||
#[doc = include_str!("../docs/TYPST.md")]
|
||||
pub mod typst_export {}
|
||||
|
||||
/// Short answers to specific questions.
|
||||
#[doc = include_str!("../docs/guide/recipes.md")]
|
||||
pub mod recipes {}
|
||||
+30
-5
@@ -26,7 +26,7 @@
|
||||
//! ## The loop
|
||||
//!
|
||||
//! ```text
|
||||
//! author items ──▶ validate ──▶ lint ──▶ assemble ──▶ export ───┐
|
||||
//! author items ──▶ validate ──▶ lint ──▶ assemble ──▶ export ──┐
|
||||
//! ▲ │
|
||||
//! │ administer
|
||||
//! │ │
|
||||
@@ -47,29 +47,54 @@
|
||||
//! already get right and then drifts from it. [`assessment::History`] derives usage
|
||||
//! by scanning the records.
|
||||
//!
|
||||
//! Fingerprints cover only what a student saw. Retag an item's metadata and its
|
||||
//! **Fingerprints cover only what a student saw.** Retag an item's metadata and its
|
||||
//! pooled statistics stay valid; reword the stem and they are marked stale. See
|
||||
//! [`item::Item::fingerprint`].
|
||||
//!
|
||||
//! Validation reports everything at once. Fixing one typo per run is not a
|
||||
//! **Validation reports everything at once.** Fixing one typo per run is not a
|
||||
//! workflow. [`error::Error::Invalid`] carries a list.
|
||||
//!
|
||||
//! Validation and linting are separate. [`bank::BankFile::validate`] enforces
|
||||
//! **Validation and linting are separate.** [`bank::BankFile::validate`] enforces
|
||||
//! what must be true; [`lint`] advises on what is usually a mistake, and every rule
|
||||
//! has a code you can silence.
|
||||
//!
|
||||
//! Small samples are labelled as such. Every statistic computed from a class of
|
||||
//! **Small samples are labelled as such.** Every statistic computed from a class of
|
||||
//! twenty-five is reported with the caveat it deserves rather than three decimal
|
||||
//! places of false precision.
|
||||
//!
|
||||
//! ## Dependency posture
|
||||
//!
|
||||
//! Deliberately small: serde, a YAML parser, clap, thiserror, and csv, plus arrow
|
||||
//! and parquet behind a default-on feature that can be switched off. Dates, PRNG,
|
||||
//! hashing, ZIP writing, and the psychometrics are implemented here rather than
|
||||
//! pulled in — see [`date`], [`rng`], [`hash`], [`zipfile`], [`irt`]. For a tool
|
||||
//! whose job is to still open a course repository in five years, that tradeoff
|
||||
//! favours fewer moving parts.
|
||||
//!
|
||||
//! ## Where to start
|
||||
//!
|
||||
//! This page describes the shape of the crate. For a walkthrough, [`guide`] holds
|
||||
//! setup, authoring, and tutorials, starting with [`guide::setup`].
|
||||
|
||||
#![warn(missing_docs)]
|
||||
#![forbid(unsafe_code)]
|
||||
// A broken link in a tutorial is a silent lie about the API, so it fails the build
|
||||
// rather than warning into a log nobody reads.
|
||||
#![deny(rustdoc::broken_intra_doc_links)]
|
||||
#![warn(rustdoc::invalid_codeblock_attributes)]
|
||||
#![warn(rustdoc::invalid_html_tags)]
|
||||
#![warn(rustdoc::bare_urls)]
|
||||
#![warn(rustdoc::private_intra_doc_links)]
|
||||
// Lets `cargo doc` mark feature-gated items with the feature that enables them, on
|
||||
// a nightly toolchain or on docs.rs. Ignored elsewhere.
|
||||
#![cfg_attr(docsrs, feature(doc_cfg))]
|
||||
|
||||
pub mod analysis;
|
||||
pub mod authoring;
|
||||
pub mod data;
|
||||
pub mod error;
|
||||
pub mod export;
|
||||
pub mod guide;
|
||||
pub mod model;
|
||||
pub mod util;
|
||||
|
||||
|
||||
+1
-1
@@ -110,7 +110,7 @@ struct InitArgs {
|
||||
/// Course title.
|
||||
#[arg(long)]
|
||||
title: String,
|
||||
/// Term, e.g. 2026S.
|
||||
/// Term, e.g. 2026s.
|
||||
#[arg(long)]
|
||||
term: String,
|
||||
/// Also write an example bank and assessment.
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
//!
|
||||
//! Grading exports do not know about your item bank. Gradescope gives you
|
||||
//! `14.csv`; Canvas gives you a column header. Both identify questions by
|
||||
//! position on a form. An item bank identifies questions by stable id. The
|
||||
//! *position on a form*. An item bank identifies questions by stable id. The
|
||||
//! assessment record is the only place those two namespaces meet, and without it
|
||||
//! there is no way to say that question 14 of Exam 4 was
|
||||
//! `docking::q-scoring-003` at version 2.
|
||||
@@ -15,7 +15,7 @@
|
||||
//! assessment records: an item was used exactly when it appears on a record. The
|
||||
//! records are the source of truth, and they are small, readable, and diffable.
|
||||
//!
|
||||
//! Each placement stores the resolved key and content fingerprint as used. A
|
||||
//! Each placement stores the resolved key and content fingerprint *as used*. A
|
||||
//! year later, when the item has been reworded twice, you can still see what the
|
||||
//! students in front of you were asked.
|
||||
|
||||
@@ -66,7 +66,7 @@ pub struct AssessmentFile {
|
||||
pub struct Assessment {
|
||||
/// Stable id, e.g. `exam-4-2026s`. Response tables carry this.
|
||||
pub id: String,
|
||||
/// Human title as printed, e.g., `Exam 4`.
|
||||
/// Human title as printed, e.g. `Exam 4`.
|
||||
pub title: String,
|
||||
/// The term this administration belongs to. Items outlive terms, so the term
|
||||
/// lives here rather than on the item.
|
||||
|
||||
+12
-12
@@ -3,13 +3,13 @@
|
||||
//! One bank per topic (or per lecture, if that suits how you teach) is the unit
|
||||
//! of authoring. Banks are small enough to review in a pull request, they let
|
||||
//! two people write questions without colliding, and `bank.scope` records what
|
||||
//! the file is for so `coursebank catalog` can tell you that you have eleven
|
||||
//! the file is *for* so `coursebank catalog` can tell you that you have eleven
|
||||
//! items on enzyme kinetics and none on regulation.
|
||||
//!
|
||||
//! Validation here is split in two on purpose. [`BankFile::validate`] checks what
|
||||
//! must be true for the file to be usable at all: ids are unique, a keyed answer
|
||||
//! exists, an approved item is fully specified, a level and its cognitive process
|
||||
//! agree. The softer question of whether an item is well written lives in
|
||||
//! agree. The softer question of whether an item is *well written* lives in
|
||||
//! [`crate::lint`], because those checks are advisory and you should be able to
|
||||
//! ship a file that trips a few of them.
|
||||
|
||||
@@ -76,7 +76,7 @@ pub struct BankMeta {
|
||||
/// What a bank is scoped to.
|
||||
///
|
||||
/// A bank may be scoped by lecture, by objective, by topic, or by none of them.
|
||||
/// Declaring the scope is what lets the catalog report gaps: it can only tell
|
||||
/// Declaring the scope is what lets the catalog report *gaps*: it can only tell
|
||||
/// you that lecture 12 has no Apply-level items if it knows lecture 12 is
|
||||
/// supposed to be covered here.
|
||||
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
|
||||
@@ -348,7 +348,7 @@ fn validate_item(
|
||||
issues.push("version must be at least 1".into());
|
||||
}
|
||||
|
||||
// --- options ---
|
||||
// --- options -----------------------------------------------------------
|
||||
if it.options.len() < 2 {
|
||||
issues.push(format!(
|
||||
"needs at least 2 options, has {}",
|
||||
@@ -414,7 +414,7 @@ fn validate_item(
|
||||
}
|
||||
}
|
||||
|
||||
// --- key ---
|
||||
// --- key ---------------------------------------------------------------
|
||||
let keys = it.key_indices();
|
||||
match it.format {
|
||||
Format::SingleBestAnswer => {
|
||||
@@ -446,7 +446,7 @@ fn validate_item(
|
||||
}
|
||||
}
|
||||
|
||||
// --- level and process must agree ---
|
||||
// --- level and process must agree -------------------------------------
|
||||
if let Some(p) = it.cognitive_process {
|
||||
if !it.level.allows(p) {
|
||||
issues.push(format!(
|
||||
@@ -457,7 +457,7 @@ fn validate_item(
|
||||
}
|
||||
}
|
||||
|
||||
// --- design plausibility ---
|
||||
// --- design plausibility ----------------------------------------------
|
||||
if let Some(d) = &it.design {
|
||||
if let Some(x) = d.expected_difficulty {
|
||||
if !(0.0..=1.0).contains(&x) {
|
||||
@@ -475,7 +475,7 @@ fn validate_item(
|
||||
}
|
||||
}
|
||||
|
||||
// --- calibration plausibility ---
|
||||
// --- calibration plausibility -----------------------------------------
|
||||
if let Some(c) = &it.calibration {
|
||||
if let Some(p) = c.p_value {
|
||||
if !(0.0..=1.0).contains(&p) {
|
||||
@@ -510,7 +510,7 @@ fn validate_item(
|
||||
}
|
||||
}
|
||||
|
||||
// --- history must be coherent ---
|
||||
// --- history must be coherent -----------------------------------------
|
||||
let mut last_version = 0u32;
|
||||
for (i, h) in it.history.iter().enumerate() {
|
||||
if h.version <= last_version {
|
||||
@@ -529,7 +529,7 @@ fn validate_item(
|
||||
));
|
||||
}
|
||||
|
||||
// --- retirement ---
|
||||
// --- retirement -------------------------------------------------------
|
||||
if it.retired.is_some() && it.status != Status::Retired {
|
||||
issues.push(format!(
|
||||
"has a `retired` block but status is `{}`",
|
||||
@@ -537,7 +537,7 @@ fn validate_item(
|
||||
));
|
||||
}
|
||||
|
||||
// --- approval gate ---
|
||||
// --- approval gate ----------------------------------------------------
|
||||
// Approval is what permits an item onto a graded assessment, so it is the
|
||||
// right place to require that the item is fully sourced and designed.
|
||||
if it.status == Status::Approved {
|
||||
@@ -555,7 +555,7 @@ fn validate_item(
|
||||
}
|
||||
}
|
||||
|
||||
// --- cross-file references ---
|
||||
// --- cross-file references --------------------------------------------
|
||||
if let Some(c) = course {
|
||||
for lo in &it.learning_objectives {
|
||||
match c.learning_objectives.get(lo) {
|
||||
|
||||
+1
-1
@@ -8,7 +8,7 @@
|
||||
//! Keeping them adjacent is what turns a question bank into an instrument you
|
||||
//! can improve, because every administration produces a checkable prediction.
|
||||
//!
|
||||
//! One deliberate departure from a naive design: [`Calibration`] is cumulative
|
||||
//! One deliberate departure from a naive design: [`Calibration`] is *cumulative*
|
||||
//! rather than per-administration. Raw per-response data belongs in the Parquet
|
||||
//! tables under `data/`, which are far better at holding it, and an item's YAML
|
||||
//! holds the rolled-up estimate plus a list of which administrations went into
|
||||
|
||||
@@ -1,5 +1,12 @@
|
||||
//! The pedagogical vocabulary: levels, cognitive processes, error types, and
|
||||
//! workflow states.
|
||||
//!
|
||||
//! These are enums rather than strings on purpose. A typo in
|
||||
//! `cognitive_process` should fail to parse, not silently create a new category
|
||||
//! that then splits your coverage report in two. Just as importantly, the
|
||||
//! relation between a level and the processes that belong to it is encoded here
|
||||
//! in one place, so "level 3, cognitive_process: recall" is a validation error
|
||||
//! rather than a label that quietly contradicts itself.
|
||||
|
||||
use std::fmt;
|
||||
|
||||
|
||||
+220
@@ -0,0 +1,220 @@
|
||||
//! Checks the markdown that gets included into the documentation.
|
||||
//!
|
||||
//! Rustdoc reads a fenced block with an empty info string as Rust and compiles it.
|
||||
//! So a `course.yaml` snippet in a bare fence fails the doc build with a parse
|
||||
//! error pointing at a generated file, which is a confusing way to learn that a
|
||||
//! four-letter language tag is missing. This test names the file and line instead.
|
||||
//!
|
||||
//! Two related problems are already covered elsewhere and are not repeated here: a
|
||||
//! missing `include_str!` target fails compilation, and a broken intra-doc link
|
||||
//! fails under `#![deny(rustdoc::broken_intra_doc_links)]` in `lib.rs`.
|
||||
//!
|
||||
//! No regex engine, matching the rest of the crate: every rule is a scan.
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
/// Info-string words rustdoc reads as attributes on a *Rust* block. Anything else
|
||||
/// is a language name, and rustdoc skips the block.
|
||||
const RUST_ATTRS: &[&str] = &[
|
||||
"rust",
|
||||
"ignore",
|
||||
"should_panic",
|
||||
"no_run",
|
||||
"compile_fail",
|
||||
"test_harness",
|
||||
"standalone_crate",
|
||||
];
|
||||
|
||||
/// Language tags this repository uses for content that is not Rust.
|
||||
const KNOWN_LANGS: &[&str] = &[
|
||||
"console", "text", "yaml", "toml", "json", "bash", "sh", "shell", "typst", "diff", "csv",
|
||||
"markdown", "md", "xml", "html", "python",
|
||||
];
|
||||
|
||||
/// The repository root, derived from the manifest directory.
|
||||
fn root() -> PathBuf {
|
||||
PathBuf::from(env!("CARGO_MANIFEST_DIR"))
|
||||
}
|
||||
|
||||
/// Every `.md` file under `docs/`, sorted so failures are reported in a stable
|
||||
/// order.
|
||||
fn markdown_files(dir: &Path, out: &mut Vec<PathBuf>) {
|
||||
let Ok(entries) = std::fs::read_dir(dir) else {
|
||||
return;
|
||||
};
|
||||
let mut paths: Vec<PathBuf> = entries.filter_map(|e| e.ok()).map(|e| e.path()).collect();
|
||||
paths.sort();
|
||||
for path in paths {
|
||||
if path.is_dir() {
|
||||
markdown_files(&path, out);
|
||||
} else if path.extension().is_some_and(|e| e == "md") {
|
||||
out.push(path);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// A fence opener: its line number, its run of backticks, and its info string.
|
||||
struct Fence<'a> {
|
||||
line: usize,
|
||||
ticks: usize,
|
||||
info: &'a str,
|
||||
}
|
||||
|
||||
/// Finds every opening fence, tracking nesting by backtick count so that a `` ``` ``
|
||||
/// inside a ```` ```` ```` block is treated as content rather than as a new fence.
|
||||
fn opening_fences(text: &str) -> (Vec<Fence<'_>>, Option<usize>) {
|
||||
let mut found = Vec::new();
|
||||
let mut open: Option<usize> = None;
|
||||
|
||||
for (index, raw) in text.lines().enumerate() {
|
||||
let line = raw.trim_start();
|
||||
if !line.starts_with("```") {
|
||||
continue;
|
||||
}
|
||||
let ticks = line.chars().take_while(|c| *c == '`').count();
|
||||
let info = line[ticks..].trim();
|
||||
|
||||
match open {
|
||||
None => {
|
||||
open = Some(ticks);
|
||||
found.push(Fence {
|
||||
line: index + 1,
|
||||
ticks,
|
||||
info,
|
||||
});
|
||||
}
|
||||
// A closing fence is at least as long as its opener and carries no info
|
||||
// string. Anything else is content inside the block.
|
||||
Some(width) if ticks >= width && info.is_empty() => open = None,
|
||||
Some(_) => {}
|
||||
}
|
||||
}
|
||||
(found, open)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn every_code_fence_declares_its_language() {
|
||||
let root = root();
|
||||
let mut files = Vec::new();
|
||||
markdown_files(&root.join("docs"), &mut files);
|
||||
assert!(!files.is_empty(), "no markdown found under docs/");
|
||||
|
||||
let mut issues: Vec<String> = Vec::new();
|
||||
|
||||
for path in &files {
|
||||
let text = std::fs::read_to_string(path).expect("markdown is readable");
|
||||
let name = path.strip_prefix(&root).unwrap_or(path).display();
|
||||
let (fences, unclosed) = opening_fences(&text);
|
||||
|
||||
if unclosed.is_some() {
|
||||
issues.push(format!(
|
||||
"{name}: a code fence is never closed. A nested ``` inside a block \
|
||||
closes its parent early; widen the outer fence to ````."
|
||||
));
|
||||
}
|
||||
|
||||
for fence in fences {
|
||||
let _ = fence.ticks;
|
||||
if fence.info.is_empty() {
|
||||
issues.push(format!(
|
||||
"{name}:{}: fence has no language, so rustdoc compiles it as Rust. \
|
||||
Tag it (```console, ```yaml, ```text) or mark it ```rust.",
|
||||
fence.line
|
||||
));
|
||||
continue;
|
||||
}
|
||||
|
||||
let words: Vec<&str> = fence
|
||||
.info
|
||||
.split(|c: char| c == ',' || c.is_whitespace())
|
||||
.filter(|w| !w.is_empty())
|
||||
.collect();
|
||||
let head = words[0];
|
||||
|
||||
if RUST_ATTRS.contains(&head) || head.starts_with("edition") {
|
||||
// A Rust block. Every further word must be a real attribute, since a
|
||||
// typo like `no_ru` silently stops the example from being tested.
|
||||
for word in &words {
|
||||
if !RUST_ATTRS.contains(word) && !word.starts_with("edition") {
|
||||
issues.push(format!(
|
||||
"{name}:{}: `{word}` is not a rustdoc code attribute, so this \
|
||||
block is silently not tested",
|
||||
fence.line
|
||||
));
|
||||
}
|
||||
}
|
||||
} else if !KNOWN_LANGS.contains(&head) {
|
||||
issues.push(format!(
|
||||
"{name}:{}: unrecognized language `{head}`; add it to KNOWN_LANGS \
|
||||
in tests/docs.rs if that is intended",
|
||||
fence.line
|
||||
));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
assert!(
|
||||
issues.is_empty(),
|
||||
"{} problem(s) in included markdown:\n{}",
|
||||
issues.len(),
|
||||
issues
|
||||
.iter()
|
||||
.map(|i| format!(" - {i}"))
|
||||
.collect::<Vec<_>>()
|
||||
.join("\n")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_guide_includes_every_markdown_file_under_docs() {
|
||||
// A page nobody includes is a page nobody reads. This catches a new file in
|
||||
// docs/ that was never wired into the module tree.
|
||||
let root = root();
|
||||
let guide = std::fs::read_to_string(root.join("src/guide.rs")).expect("src/guide.rs exists");
|
||||
let lib = std::fs::read_to_string(root.join("src/lib.rs")).expect("src/lib.rs exists");
|
||||
let included = format!("{guide}{lib}");
|
||||
|
||||
let mut files = Vec::new();
|
||||
markdown_files(&root.join("docs"), &mut files);
|
||||
|
||||
let orphans: Vec<String> = files
|
||||
.iter()
|
||||
.filter(|path| {
|
||||
let name = path.file_name().unwrap().to_string_lossy().to_string();
|
||||
!included.contains(&name)
|
||||
})
|
||||
.map(|path| {
|
||||
path.strip_prefix(&root)
|
||||
.unwrap_or(path)
|
||||
.display()
|
||||
.to_string()
|
||||
})
|
||||
.collect();
|
||||
|
||||
assert!(
|
||||
orphans.is_empty(),
|
||||
"these files are not included by any doc attribute:\n{}\nAdd a doc-only module \
|
||||
in src/guide.rs, or move the file out of docs/.",
|
||||
orphans
|
||||
.iter()
|
||||
.map(|o| format!(" - {o}"))
|
||||
.collect::<Vec<_>>()
|
||||
.join("\n")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn fence_scanning_handles_nesting_and_tags() {
|
||||
let (fences, unclosed) = opening_fences("```yaml\na: 1\n```\n");
|
||||
assert_eq!(fences.len(), 1);
|
||||
assert_eq!(fences[0].info, "yaml");
|
||||
assert!(unclosed.is_none());
|
||||
|
||||
// A ``` inside a ```` block is content, not a fence.
|
||||
let (fences, unclosed) = opening_fences("````yaml\nbody: |\n ```\n table\n ```\n````\n");
|
||||
assert_eq!(fences.len(), 1, "the inner fence should not open a block");
|
||||
assert!(unclosed.is_none());
|
||||
|
||||
let (_, unclosed) = opening_fences("```text\nno end\n");
|
||||
assert!(unclosed.is_some());
|
||||
}
|
||||
Reference in New Issue
Block a user