A gate that does not depend on its machine, on a runner of its own
Test / gate (push) Canceled after 0s

The gate first ran on the server that deploys the site: a slow container
with no C compiler and no ICU. Ten tests failed there on correct code,
and no Mojo program could be built at all.

- The workflows ask for a runner labeled heavy, not the deploy server:
  the gate builds and runs some 300 Mojo programs and fuzzes. docs/ci.md
  says how to register one. With none registered a run waits.
- The pinned environment has a C compiler on Linux (gcc, with its C
  library pinned to glibc 2.34): mojo build links with the C compiler it
  finds, and conda-forge's default links against a C library too old
  for Mojo's runtime. mojo-local reports a program that could not be
  linked as that, and does not keep the answer.
- The pinned Dafny runs without ICU, and dafny-fetch fails when the
  Dafny it fetched does not start.
- A test's time limit is processor time at the machine's pace
  (tests/helpers/pace.mjs).
- The prover's time on the clock is the caller's: provePseudo and
  Workspace.proofBudgetMs take it, the Prove tab's 3 seconds stay the
  default, and the command line, library-check and the tests use none,
  so their answers are the counted budgets' alone.
- The library pointer moves to its workflow change (e5ae090).
This commit is contained in:
alexm committed 2026-11-13 06:13:28 -05:00
1 parent 401084bab3
commit b10a5c87a3
31 files changed
+427 -104

No files matched your search

+24 -10
View File
@@ -1,24 +1,38 @@
# CI
CI runs what a developer runs. Every workflow installs pixi, restores the environment `pixi.lock` pins (Node, Python, the Mojo compiler) with `pixi install --locked`, and runs a pixi task, so the gate on the runner and the gate on a laptop are the same command in the same environment. Five weeks once passed with `main` broken in two places and nobody told: CI then ran `npm ci && npm test` with whatever Node the host had, no pandoc, no Mojo compiler and no Dafny, and a missing tool skipped tests in silence.
CI runs what a developer runs. Every workflow installs pixi, restores the environment `pixi.lock` pins (Node, Python, the Mojo compiler, and a C compiler on Linux) with `pixi install --locked`, and runs a pixi task, so the gate on the runner and the gate on a laptop are the same command in the same environment. Five weeks once passed with `main` broken in two places and nobody told: CI then ran `npm ci && npm test` with whatever Node the host had, no pandoc, no Mojo compiler and no Dafny, and a missing tool skipped tests in silence.
| Workflow | When | Runs | Time here |
|---|---|---|---|
| `.gitea/workflows/test.yml` | every push and pull request | `pixi run ci`, then checks that nothing the gate regenerates differs from what is committed | 4 minutes here, 2.5 with `mojo-local`'s answers cached |
| `.gitea/workflows/nightly.yml` | every day at 06:17 UTC, and by hand | `pixi run nightly` | 12 minutes here |
| pplus-library's `.gitea/workflows/test.yml` | every push and pull request to the library | core's default branch with that library commit in `library/`, then `pixi run ci` (below) | as core's |
| `.gitea/workflows/test.yml` | every push and pull request, on the `heavy` runner | `pixi run ci`, then checks that nothing the gate regenerates differs from what is committed | 4 minutes here, 2.5 with `mojo-local`'s answers cached |
| `.gitea/workflows/nightly.yml` | every day at 06:17 UTC, and by hand, on the `heavy` runner | `pixi run nightly` | 12 minutes here |
| pplus-library's `.gitea/workflows/test.yml` | every push and pull request to the library, on the `heavy` runner | core's default branch with that library commit in `library/`, then `pixi run ci` (below) | as core's |
`pixi run ci` is `test` (every test), `mojo-status-strict` (no Mojo program of ours fails, from godbolt's stored answers), `library-check` (every library routine at every level, Dafny included), `mojo-local` (every Mojo program core holds or generates, built and run by the pinned compiler) and `ci-fuzz` (below). `pixi run nightly` is `mojo-status-strict` and the fuzzers for longer.
## The runner
The workflows are written for Gitea Actions on a runner in host mode (no container: steps run on the host's shell); pplus-web's `docs/deploy.md` has the setup. They use no `container:` and no `services:`, and each needs one change to run on GitHub Actions: `runs-on: ubuntu-latest` in place of the host's label.
The gate is heavy: it builds and runs some 300 Mojo programs, checks every library routine and fuzzes, about 4 minutes on a 24-core workstation, and 4 and a half on two cores in the deploy runner's image with nothing cached. So the workflows run on a runner with the label `heavy`, a machine with cores to spare, and never on the server that deploys the site (pplus-web's deploy job, label `quarto-deploy`, runs the site's own tests and nothing of this gate). While no `heavy` runner is registered a run waits, and the next push's run replaces it; until one is, the gate is `pixi run ci` on a developer's machine before a push.
- **pixi.** `PIXI_VERSION` at the top of each workflow is the pixi that reads the lock. The first step uses the host's pixi when it is that version; otherwise it downloads that release from GitHub into `$PPLUS_CI_CACHE/pixi-<version>/` (default `~/.cache/pplus-ci`), checks its SHA-256 (written in the workflow), and puts it on the PATH. No sudo; a host runner fetches it once. To move pixi: change `PIXI_VERSION` and both hashes in every workflow, in one commit, and check that `pixi install --locked` still accepts the lock.
- **The environment.** `pixi install --locked` fails if `pixi.toml` and `pixi.lock` disagree, so the runner never resolves anything new. pixi's package cache stays on the host, so a host runner downloads each package once.
- **`.tools/`.** The pinned Dafny (`pixi run dafny-fetch`, about 150 MB) and `mojo-local`'s answers live in `.tools/`; in CI `.tools` is a link to `$PPLUS_CI_CACHE/core-tools`, so neither is fetched or rebuilt on every push. `mojo-local` keys an answer by the compiler's version, the arguments, the input and the whole source, so a cached answer is the compiler's answer for exactly that program. Two jobs that share the cache at once could write the same answer file together; the runner takes one job at a time unless its `capacity` is raised.
- **`CI`.** Each workflow sets `CI=true` (Gitea and GitHub set it too). Under it a test that would skip for a missing tool fails instead, because a skip in CI is how a gap hides: python3 (`tests/mojo.test.mjs`, `tests/dafny.test.mjs`, `tests/library.test.mjs`), the fetched Dafny (`tests/dafny.test.mjs`), a git checkout to read the library pointer from (`tests/chain.test.mjs`), and in pplus-web pandoc for the Quarto tests.
- **What the host needs besides pixi:** git, curl, tar, `sha256sum`, bash, and Node for `actions/checkout` itself; a C compiler (`cc`) for the C references `library-check` builds. Without `cc` those checks are recorded as skipped, `library/status.json` then differs from the committed one, and the last step of `test.yml` fails and says so.
Registering one, on the machine that will run it (a Gitea act_runner, as its documentation describes: download it, `act_runner generate-config > config.yaml`, run it as a service under a user of its own):
```bash
# in a container, the image the deploy runner uses (anything with git, Node, curl, tar and bash will do):
act_runner register --instance https://<your-gitea> --token <runner token> --no-interactive \
--labels "heavy:docker://<your-gitea-host>/infra/quarto-deploy:1.9.38"
# or on the machine's own shell (host mode), where the caches below last from job to job:
act_runner register --instance https://<your-gitea> --token <runner token> --no-interactive --labels "heavy:host"
```
Take the token from the organization's runner settings, so that pplus-core and pplus-library both see the runner. In `config.yaml`, `runner.capacity` is the number of jobs at once: leave it at 1 unless each job has a cache directory of its own (`.tools/`, below). The workflows use no `container:` and no `services:` of their own, and each needs one change to run on GitHub Actions: `runs-on: ubuntu-latest` in place of the label. A pull request from a fork would run its own code on that machine; each job's `if:` runs pull requests only from branches of the repository itself.
- **pixi.** `PIXI_VERSION` at the top of each workflow is the pixi that reads the lock. The first step uses the runner's pixi when it is that version; otherwise it downloads that release from GitHub into `$PPLUS_CI_CACHE/pixi-<version>/` (default `~/.cache/pplus-ci`), checks its SHA-256 (written in the workflow), and puts it on the PATH. No sudo. A runner that keeps its home between jobs (host mode) fetches it once; a container fetches it in every job. To move pixi: change `PIXI_VERSION` and both hashes in every workflow, in one commit, and check that `pixi install --locked` still accepts the lock.
- **The environment.** `pixi install --locked` fails if `pixi.toml` and `pixi.lock` disagree, so the runner never resolves anything new. On Linux the environment has a C compiler (`gcc` from conda-forge, which is also `cc` on the PATH), because `mojo build` links every program with the C compiler it finds and stops with "unable to find suitable c compiler for linking" when there is none, as in a slim container image; `library-check` builds the C references with it and one test a C harness. The C library it links against is pinned too (`sysroot_linux-64` 2.34): Mojo 1.2.0's runtime libraries need glibc 2.34, and against conda-forge's default, 2.28, no Mojo program links. So under `pixi run` it is this compiler that links, on a workstation as on the runner, whatever the system has. Where pixi's package cache lasts, each package is downloaded once; a container downloads the whole environment in every job.
- **`.tools/`.** The pinned Dafny (`pixi run dafny-fetch`, about 150 MB) and `mojo-local`'s answers live in `.tools/`; in CI `.tools` is a link to `$PPLUS_CI_CACHE/core-tools`, so where that directory lasts neither is fetched or rebuilt on every push. `mojo-local` keys an answer by the compiler's version, the arguments, the input and the whole source, so a cached answer is the compiler's answer for exactly that program. Two jobs that share the cache at once could write the same answer file together; the runner takes one job at a time unless its `capacity` is raised.
- **Dafny without ICU.** The release zip carries its own .NET runtime but not ICU, and .NET aborts without it unless told not to look: in the deploy runner's image, which has none, the fetched Dafny printed no version and aborted. `scripts/dafny.py` (and `scripts/library.mjs` and `tests/dafny.test.mjs`, which run the executable themselves) set `DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1`, and `dafny-fetch` fails if the Dafny it fetched does not start.
- **`CI`.** Each workflow sets `CI=true` (Gitea and GitHub set it too). Under it a test that would skip for a missing tool fails instead, because a skip in CI is how a gap hides: python3 (`tests/mojo.test.mjs`, `tests/dafny.test.mjs`, `tests/library.test.mjs`), the fetched Dafny (`tests/dafny.test.mjs`), `cc` (`tests/library-harness.test.mjs`), a git checkout to read the library pointer from (`tests/chain.test.mjs`), and in pplus-web pandoc for the Quarto tests.
- **What the runner needs besides pixi:** git, curl, tar, `sha256sum`, bash, and Node for `actions/checkout` itself. Everything else is pinned here: the compilers by `pixi.lock`, Dafny by its hash.
- **A slow or busy machine changes how long the gate takes, not what it finds.** The gate first ran on the deploy server, a small virtual machine: the tests that take 20 seconds on a workstation took 15 to 22 minutes there, work that takes 5 milliseconds took more than 3 seconds, and ten tests failed on correct code. Two things in the tests depended on speed then and do not now. A time limit in a test ("this reads in under 2 seconds", there to catch work that grows with the square of its input) is processor time at the machine's pace: `tests/helpers/pace.mjs` times a fixed piece of work and multiplies the limit by how much longer it takes than on the machine the limits were written on, never by less than 1. And the prover's time on the clock, 3 seconds for each routine in the Prove tab, is the caller's: the tests, `pixi run cli prove` and `library-check` give it none, so the counted budgets decide alone and the answer is the same everywhere ([limits.md](limits.md#proving-and-testing)). The fuzzers keep their clock (`--budget-ms`, below): on a slower machine `prove-fuzz` proves less, never more.
## The fuzzers' seeds