# CI CI runs what a developer runs. Every workflow installs pixi, restores the environment `pixi.lock` pins (Node, Python, the Mojo compiler, and a C compiler on Linux) with `pixi install --locked`, and runs a pixi task, so the gate on the runner and the gate on a laptop are the same command in the same environment. Five weeks once passed with `main` broken in two places and nobody told: CI then ran `npm ci && npm test` with whatever Node the host had, no pandoc, no Mojo compiler and no Dafny, and a missing tool skipped tests in silence. | Workflow | When | Runs | Time here | |---|---|---|---| | `.gitea/workflows/test.yml` | every push and pull request, on the `heavy` runner | `pixi run ci`, then checks that nothing the gate regenerates differs from what is committed | 4 minutes here, 2.5 with `mojo-local`'s answers cached | | `.gitea/workflows/nightly.yml` | every day at 06:17 UTC, and by hand, on the `heavy` runner | `pixi run nightly` | 12 minutes here | | pplus-library's `.gitea/workflows/test.yml` | every push and pull request to the library, on the `heavy` runner | core's default branch with that library commit in `library/`, then `pixi run ci` (below) | as core's | `pixi run ci` is `test` (every test), `mojo-status-strict` (no Mojo program of ours fails, from godbolt's stored answers), `library-check` (every library routine at every level, Dafny included), `mojo-local` (every Mojo program core holds or generates, built and run by the pinned compiler) and `ci-fuzz` (below). `pixi run nightly` is `mojo-status-strict` and the fuzzers for longer. ## The runner The gate is heavy: it builds and runs some 300 Mojo programs, checks every library routine and fuzzes, about 4 minutes on a 24-core workstation, and 4 and a half on two cores in the deploy runner's image with nothing cached. So the workflows run on a runner with the label `heavy`, a machine with cores to spare, and never on the server that deploys the site (pplus-web's deploy job, label `quarto-deploy`, runs the site's own tests and nothing of this gate). While no `heavy` runner is registered a run waits, and the next push's run replaces it; until one is, the gate is `pixi run ci` on a developer's machine before a push. Registering one, on the machine that will run it (a Gitea act_runner, as its documentation describes: download it, `act_runner generate-config > config.yaml`, run it as a service under a user of its own): ```bash # in a container, the image the deploy runner uses (anything with git, Node, curl, tar and bash will do): act_runner register --instance https:// --token --no-interactive \ --labels "heavy:docker:///infra/quarto-deploy:1.9.38" # or on the machine's own shell (host mode), where the caches below last from job to job: act_runner register --instance https:// --token --no-interactive --labels "heavy:host" ``` Take the token from the organization's runner settings, so that pplus-core and pplus-library both see the runner. In `config.yaml`, `runner.capacity` is the number of jobs at once: leave it at 1 unless each job has a cache directory of its own (`.tools/`, below). The workflows use no `container:` and no `services:` of their own, and each needs one change to run on GitHub Actions: `runs-on: ubuntu-latest` in place of the label. A pull request from a fork would run its own code on that machine; each job's `if:` runs pull requests only from branches of the repository itself. - **pixi.** `PIXI_VERSION` at the top of each workflow is the pixi that reads the lock. The first step uses the runner's pixi when it is that version; otherwise it downloads that release from GitHub into `$PPLUS_CI_CACHE/pixi-/` (default `~/.cache/pplus-ci`), checks its SHA-256 (written in the workflow), and puts it on the PATH. No sudo. A runner that keeps its home between jobs (host mode) fetches it once; a container fetches it in every job. To move pixi: change `PIXI_VERSION` and both hashes in every workflow, in one commit, and check that `pixi install --locked` still accepts the lock. - **The environment.** `pixi install --locked` fails if `pixi.toml` and `pixi.lock` disagree, so the runner never resolves anything new. On Linux the environment has a C compiler (`gcc` from conda-forge, which is also `cc` on the PATH), because `mojo build` links every program with the C compiler it finds and stops with "unable to find suitable c compiler for linking" when there is none, as in a slim container image; `library-check` builds the C references with it and one test a C harness. The C library it links against is pinned too (`sysroot_linux-64` 2.34): Mojo 1.2.0's runtime libraries need glibc 2.34, and against conda-forge's default, 2.28, no Mojo program links. So under `pixi run` it is this compiler that links, on a workstation as on the runner, whatever the system has. Where pixi's package cache lasts, each package is downloaded once; a container downloads the whole environment in every job. - **`.tools/`.** The pinned Dafny (`pixi run dafny-fetch`, about 150 MB) and `mojo-local`'s answers live in `.tools/`; in CI `.tools` is a link to `$PPLUS_CI_CACHE/core-tools`, so where that directory lasts neither is fetched or rebuilt on every push. `mojo-local` keys an answer by the compiler's version, the arguments, the input and the whole source, so a cached answer is the compiler's answer for exactly that program. Two jobs that share the cache at once could write the same answer file together; the runner takes one job at a time unless its `capacity` is raised. - **Dafny without ICU.** The release zip carries its own .NET runtime but not ICU, and .NET aborts without it unless told not to look: in the deploy runner's image, which has none, the fetched Dafny printed no version and aborted. `scripts/dafny.py` (and `scripts/library.mjs` and `tests/dafny.test.mjs`, which run the executable themselves) set `DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1`, and `dafny-fetch` fails if the Dafny it fetched does not start. - **`CI`.** Each workflow sets `CI=true` (Gitea and GitHub set it too). Under it a test that would skip for a missing tool fails instead, because a skip in CI is how a gap hides: python3 (`tests/mojo.test.mjs`, `tests/dafny.test.mjs`, `tests/library.test.mjs`), the fetched Dafny (`tests/dafny.test.mjs`), `cc` (`tests/library-harness.test.mjs`), a git checkout to read the library pointer from (`tests/chain.test.mjs`), and in pplus-web pandoc for the Quarto tests. - **What the runner needs besides pixi:** git, curl, tar, `sha256sum`, bash, and Node for `actions/checkout` itself. Everything else is pinned here: the compilers by `pixi.lock`, Dafny by its hash. - **A slow or busy machine changes how long the gate takes, not what it finds.** The gate first ran on the deploy server, a small virtual machine: the tests that take 20 seconds on a workstation took 15 to 22 minutes there, work that takes 5 milliseconds took more than 3 seconds, and ten tests failed on correct code. Two things in the tests depended on speed then and do not now. A time limit in a test ("this reads in under 2 seconds", there to catch work that grows with the square of its input) is processor time at the machine's pace: `tests/helpers/pace.mjs` times a fixed piece of work and multiplies the limit by how much longer it takes than on the machine the limits were written on, never by less than 1. And the prover's time on the clock, 3 seconds for each routine in the Prove tab, is the caller's: the tests, `pixi run cli prove` and `library-check` give it none, so the counted budgets decide alone and the answer is the same everywhere ([limits.md](limits.md#proving-and-testing)). The fuzzers keep their clock (`--budget-ms`, below): on a slower machine `prove-fuzz` proves less, never more. ## The fuzzers' seeds `pixi run ci-fuzz` runs `mojo-fuzz` (25 programs of expressions), `mojo-fuzz --statements` (25 whole programs) and `prove-fuzz` (300 programs) on two seeds each: seed 1, so a failure that comes back is the same failure, and a seed that moves, so every run looks somewhere new: the CI run's number (`GITHUB_RUN_NUMBER`, which Gitea sets too) or, outside CI, today's date in UTC as `YYYYMMDD`. `pixi run nightly` runs all three on four seeds, the date's number and the three numbers after it, with 100, 100 and 3,000 programs. The size matters: the statement mode's first run at 100 programs a seed found a translator fault (an argument evaluated once per pass of a loop) that many seeds of 40 had not. Every round prints its seed and the command that reproduces it: ``` == prove-fuzz seed 20261113 (reproduce: node scripts/prove-fuzz.mjs --seed 20261113 --programs 3000) ``` and the run ends with a summary, one line per round, `clean`, `FOUND` or `FAILED`. ## When CI or the nightly run fails 1. **Find the round.** The summary at the end of the log names it, with its command. 2. **Reproduce it here.** Run the same command as a pixi task, with the same seed and size: `pixi run prove-fuzz --seed 20261113 --programs 3000`, or `pixi run mojo-fuzz --seed 20261113 --programs 100`. The same seed makes the same programs on every machine. The prover has a time budget for each routine (`--budget-ms`, 1,500 ms), so a slower machine may prove less than the runner did, never more; if a prove-fuzz finding does not come back, raise the budget. 3. **A prove-fuzz finding** is a soundness bug: an obligation marked proved that a run breaks. The report names the program's number and prints the smallest program that still shows it, with the call and the run's seed. `pixi run prove-fuzz --seed S --only-program N --keep work/` writes it as a regression program with its `// CALL:` and `// MUST-NOT-PROVE:` header. Check it by hand (`pixi run cli prove ` says proved; running it with that call breaks it), put it in `library/pseudo/prover/`, watch `tests/prover-soundness.test.mjs` fail, then fix the rule (AGENTS.md, rule 5; [engines.md](engines.md#soundness)). `--without ` switches off one construct at a time to see which one the finding needs. 4. **A mojo-fuzz finding** is printed once per shape with the smallest expression that shows it; its header (`scripts/mojo/fuzz.mjs`) says what each kind means. A translator fix is pinned in `tests/sync-mojo.test.mjs`; a finding that is the compiler's own is listed there as such, with what was observed. 5. **mojo-status-strict** failing means a program we wrote fails against godbolt's stored answers: see [mojo-corpus.md](mojo-corpus.md). 6. **Anything else** in `pixi run ci` fails the same way on a developer's machine: run `pixi run ci` and read the first failure. A failure opens no issue and sends nothing: the run is red on the repository's Actions page, for whoever looks. Look after a push, and look at the nightly runs at least weekly. ## The pointer to the library Core pins pplus-library as `library/` by commit. `tests/chain.test.mjs` fails when that commit is not in the library checkout, or the checkout is behind it, and says which command fixes it (`git submodule update --init`). The whole chain (web pins core, core pins library; each pointer pushed to its branch; the generated files current; Mojo pinned alike) is checked by pplus-web's `release-check` task, and moved in order by its `release-bump` task (pplus-web's `docs/deploy.md`). pplus-library has no tests of its own: its checks are core's. Its workflow checks out core's default branch (or the ref in the repository variable `CORE_REF`), puts the library commit being tested where core's submodule goes, moves core's pointer to it in that throwaway checkout and regenerates what core derives from the library (`library-emit`, `mojo-emit`), and runs `pixi run ci`. A green run shows that core's default branch, given this library and what core regenerates from it, passes core's whole gate. It does not show that core pins this commit (that is a later commit in core), that the site works with it, or anything about a core branch other than the one tested; and a red run can be core's own, so look at core's latest run first.