# CI CI runs what a developer runs. Every workflow installs pixi, restores the environment `pixi.lock` pins (Node, Python, the Mojo compiler) with `pixi install --locked`, and runs a pixi task, so the gate on the runner and the gate on a laptop are the same command in the same environment. Five weeks once passed with `main` broken in two places and nobody told: CI then ran `npm ci && npm test` with whatever Node the host had, no pandoc, no Mojo compiler and no Dafny, and a missing tool skipped tests in silence. | Workflow | When | Runs | Time here | |---|---|---|---| | `.gitea/workflows/test.yml` | every push and pull request | `pixi run ci`, then checks that nothing the gate regenerates differs from what is committed | 4 minutes here, 2.5 with `mojo-local`'s answers cached | | `.gitea/workflows/nightly.yml` | every day at 06:17 UTC, and by hand | `pixi run nightly` | 12 minutes here | | pplus-library's `.gitea/workflows/test.yml` | every push and pull request to the library | core's default branch with that library commit in `library/`, then `pixi run ci` (below) | as core's | `pixi run ci` is `test` (every test), `mojo-status-strict` (no Mojo program of ours fails, from godbolt's stored answers), `library-check` (every library routine at every level, Dafny included), `mojo-local` (every Mojo program core holds or generates, built and run by the pinned compiler) and `ci-fuzz` (below). `pixi run nightly` is `mojo-status-strict` and the fuzzers for longer. ## The runner The workflows are written for Gitea Actions on a runner in host mode (no container: steps run on the host's shell); pplus-web's `docs/deploy.md` has the setup. They use no `container:` and no `services:`, and each needs one change to run on GitHub Actions: `runs-on: ubuntu-latest` in place of the host's label. - **pixi.** `PIXI_VERSION` at the top of each workflow is the pixi that reads the lock. The first step uses the host's pixi when it is that version; otherwise it downloads that release from GitHub into `$PPLUS_CI_CACHE/pixi-/` (default `~/.cache/pplus-ci`), checks its SHA-256 (written in the workflow), and puts it on the PATH. No sudo; a host runner fetches it once. To move pixi: change `PIXI_VERSION` and both hashes in every workflow, in one commit, and check that `pixi install --locked` still accepts the lock. - **The environment.** `pixi install --locked` fails if `pixi.toml` and `pixi.lock` disagree, so the runner never resolves anything new. pixi's package cache stays on the host, so a host runner downloads each package once. - **`.tools/`.** The pinned Dafny (`pixi run dafny-fetch`, about 150 MB) and `mojo-local`'s answers live in `.tools/`; in CI `.tools` is a link to `$PPLUS_CI_CACHE/core-tools`, so neither is fetched or rebuilt on every push. `mojo-local` keys an answer by the compiler's version, the arguments, the input and the whole source, so a cached answer is the compiler's answer for exactly that program. Two jobs that share the cache at once could write the same answer file together; the runner takes one job at a time unless its `capacity` is raised. - **`CI`.** Each workflow sets `CI=true` (Gitea and GitHub set it too). Under it a test that would skip for a missing tool fails instead, because a skip in CI is how a gap hides: python3 (`tests/mojo.test.mjs`, `tests/dafny.test.mjs`, `tests/library.test.mjs`), the fetched Dafny (`tests/dafny.test.mjs`), a git checkout to read the library pointer from (`tests/chain.test.mjs`), and in pplus-web pandoc for the Quarto tests. - **What the host needs besides pixi:** git, curl, tar, `sha256sum`, bash, and Node for `actions/checkout` itself; a C compiler (`cc`) for the C references `library-check` builds. Without `cc` those checks are recorded as skipped, `library/status.json` then differs from the committed one, and the last step of `test.yml` fails and says so. ## The fuzzers' seeds `pixi run ci-fuzz` runs `mojo-fuzz` (25 programs of expressions), `mojo-fuzz --statements` (25 whole programs) and `prove-fuzz` (300 programs) on two seeds each: seed 1, so a failure that comes back is the same failure, and a seed that moves, so every run looks somewhere new: the CI run's number (`GITHUB_RUN_NUMBER`, which Gitea sets too) or, outside CI, today's date in UTC as `YYYYMMDD`. `pixi run nightly` runs all three on four seeds, the date's number and the three numbers after it, with 100, 100 and 3,000 programs. The size matters: the statement mode's first run at 100 programs a seed found a translator fault (an argument evaluated once per pass of a loop) that many seeds of 40 had not. Every round prints its seed and the command that reproduces it: ``` == prove-fuzz seed 20261113 (reproduce: node scripts/prove-fuzz.mjs --seed 20261113 --programs 3000) ``` and the run ends with a summary, one line per round, `clean`, `FOUND` or `FAILED`. ## When CI or the nightly run fails 1. **Find the round.** The summary at the end of the log names it, with its command. 2. **Reproduce it here.** Run the same command as a pixi task, with the same seed and size: `pixi run prove-fuzz --seed 20261113 --programs 3000`, or `pixi run mojo-fuzz --seed 20261113 --programs 100`. The same seed makes the same programs on every machine. The prover has a time budget for each routine (`--budget-ms`, 1,500 ms), so a slower machine may prove less than the runner did, never more; if a prove-fuzz finding does not come back, raise the budget. 3. **A prove-fuzz finding** is a soundness bug: an obligation marked proved that a run breaks. The report names the program's number and prints the smallest program that still shows it, with the call and the run's seed. `pixi run prove-fuzz --seed S --only-program N --keep work/` writes it as a regression program with its `// CALL:` and `// MUST-NOT-PROVE:` header. Check it by hand (`pixi run cli prove ` says proved; running it with that call breaks it), put it in `library/pseudo/prover/`, watch `tests/prover-soundness.test.mjs` fail, then fix the rule (AGENTS.md, rule 5; [engines.md](engines.md#soundness)). `--without ` switches off one construct at a time to see which one the finding needs. 4. **A mojo-fuzz finding** is printed once per shape with the smallest expression that shows it; its header (`scripts/mojo/fuzz.mjs`) says what each kind means. A translator fix is pinned in `tests/sync-mojo.test.mjs`; a finding that is the compiler's own is listed there as such, with what was observed. 5. **mojo-status-strict** failing means a program we wrote fails against godbolt's stored answers: see [mojo-corpus.md](mojo-corpus.md). 6. **Anything else** in `pixi run ci` fails the same way on a developer's machine: run `pixi run ci` and read the first failure. A failure opens no issue and sends nothing: the run is red on the repository's Actions page, for whoever looks. Look after a push, and look at the nightly runs at least weekly. ## The pointer to the library Core pins pplus-library as `library/` by commit. `tests/chain.test.mjs` fails when that commit is not in the library checkout, or the checkout is behind it, and says which command fixes it (`git submodule update --init`). The whole chain (web pins core, core pins library; each pointer pushed to its branch; the generated files current; Mojo pinned alike) is checked by pplus-web's `release-check` task, and moved in order by its `release-bump` task (pplus-web's `docs/deploy.md`). pplus-library has no tests of its own: its checks are core's. Its workflow checks out core's default branch (or the ref in the repository variable `CORE_REF`), puts the library commit being tested where core's submodule goes, moves core's pointer to it in that throwaway checkout and regenerates what core derives from the library (`library-emit`, `mojo-emit`), and runs `pixi run ci`. A green run shows that core's default branch, given this library and what core regenerates from it, passes core's whole gate. It does not show that core pins this commit (that is a later commit in core), that the site works with it, or anything about a core branch other than the one tested; and a red run can be core's own, so look at core's latest run first.