A gate that does not depend on its machine, on a runner of its own
Test / gate (push) Canceled after 0s
Test / gate (push) Canceled after 0s
The gate first ran on the server that deploys the site: a slow container with no C compiler and no ICU. Ten tests failed there on correct code, and no Mojo program could be built at all. - The workflows ask for a runner labeled heavy, not the deploy server: the gate builds and runs some 300 Mojo programs and fuzzes. docs/ci.md says how to register one. With none registered a run waits. - The pinned environment has a C compiler on Linux (gcc, with its C library pinned to glibc 2.34): mojo build links with the C compiler it finds, and conda-forge's default links against a C library too old for Mojo's runtime. mojo-local reports a program that could not be linked as that, and does not keep the answer. - The pinned Dafny runs without ICU, and dafny-fetch fails when the Dafny it fetched does not start. - A test's time limit is processor time at the machine's pace (tests/helpers/pace.mjs). - The prover's time on the clock is the caller's: provePseudo and Workspace.proofBudgetMs take it, the Prove tab's 3 seconds stay the default, and the command line, library-check and the tests use none, so their answers are the counted budgets' alone. - The library pointer moves to its workflow change (e5ae090).
This commit is contained in:
31 files changed
+427
-104
No files matched your search
+24
-10
@@ -1,24 +1,38 @@
|
||||
# CI
|
||||
|
||||
CI runs what a developer runs. Every workflow installs pixi, restores the environment `pixi.lock` pins (Node, Python, the Mojo compiler) with `pixi install --locked`, and runs a pixi task, so the gate on the runner and the gate on a laptop are the same command in the same environment. Five weeks once passed with `main` broken in two places and nobody told: CI then ran `npm ci && npm test` with whatever Node the host had, no pandoc, no Mojo compiler and no Dafny, and a missing tool skipped tests in silence.
|
||||
CI runs what a developer runs. Every workflow installs pixi, restores the environment `pixi.lock` pins (Node, Python, the Mojo compiler, and a C compiler on Linux) with `pixi install --locked`, and runs a pixi task, so the gate on the runner and the gate on a laptop are the same command in the same environment. Five weeks once passed with `main` broken in two places and nobody told: CI then ran `npm ci && npm test` with whatever Node the host had, no pandoc, no Mojo compiler and no Dafny, and a missing tool skipped tests in silence.
|
||||
|
||||
| Workflow | When | Runs | Time here |
|
||||
|---|---|---|---|
|
||||
| `.gitea/workflows/test.yml` | every push and pull request | `pixi run ci`, then checks that nothing the gate regenerates differs from what is committed | 4 minutes here, 2.5 with `mojo-local`'s answers cached |
|
||||
| `.gitea/workflows/nightly.yml` | every day at 06:17 UTC, and by hand | `pixi run nightly` | 12 minutes here |
|
||||
| pplus-library's `.gitea/workflows/test.yml` | every push and pull request to the library | core's default branch with that library commit in `library/`, then `pixi run ci` (below) | as core's |
|
||||
| `.gitea/workflows/test.yml` | every push and pull request, on the `heavy` runner | `pixi run ci`, then checks that nothing the gate regenerates differs from what is committed | 4 minutes here, 2.5 with `mojo-local`'s answers cached |
|
||||
| `.gitea/workflows/nightly.yml` | every day at 06:17 UTC, and by hand, on the `heavy` runner | `pixi run nightly` | 12 minutes here |
|
||||
| pplus-library's `.gitea/workflows/test.yml` | every push and pull request to the library, on the `heavy` runner | core's default branch with that library commit in `library/`, then `pixi run ci` (below) | as core's |
|
||||
|
||||
`pixi run ci` is `test` (every test), `mojo-status-strict` (no Mojo program of ours fails, from godbolt's stored answers), `library-check` (every library routine at every level, Dafny included), `mojo-local` (every Mojo program core holds or generates, built and run by the pinned compiler) and `ci-fuzz` (below). `pixi run nightly` is `mojo-status-strict` and the fuzzers for longer.
|
||||
|
||||
## The runner
|
||||
|
||||
The workflows are written for Gitea Actions on a runner in host mode (no container: steps run on the host's shell); pplus-web's `docs/deploy.md` has the setup. They use no `container:` and no `services:`, and each needs one change to run on GitHub Actions: `runs-on: ubuntu-latest` in place of the host's label.
|
||||
The gate is heavy: it builds and runs some 300 Mojo programs, checks every library routine and fuzzes, about 4 minutes on a 24-core workstation, and 4 and a half on two cores in the deploy runner's image with nothing cached. So the workflows run on a runner with the label `heavy`, a machine with cores to spare, and never on the server that deploys the site (pplus-web's deploy job, label `quarto-deploy`, runs the site's own tests and nothing of this gate). While no `heavy` runner is registered a run waits, and the next push's run replaces it; until one is, the gate is `pixi run ci` on a developer's machine before a push.
|
||||
|
||||
- **pixi.** `PIXI_VERSION` at the top of each workflow is the pixi that reads the lock. The first step uses the host's pixi when it is that version; otherwise it downloads that release from GitHub into `$PPLUS_CI_CACHE/pixi-<version>/` (default `~/.cache/pplus-ci`), checks its SHA-256 (written in the workflow), and puts it on the PATH. No sudo; a host runner fetches it once. To move pixi: change `PIXI_VERSION` and both hashes in every workflow, in one commit, and check that `pixi install --locked` still accepts the lock.
|
||||
- **The environment.** `pixi install --locked` fails if `pixi.toml` and `pixi.lock` disagree, so the runner never resolves anything new. pixi's package cache stays on the host, so a host runner downloads each package once.
|
||||
- **`.tools/`.** The pinned Dafny (`pixi run dafny-fetch`, about 150 MB) and `mojo-local`'s answers live in `.tools/`; in CI `.tools` is a link to `$PPLUS_CI_CACHE/core-tools`, so neither is fetched or rebuilt on every push. `mojo-local` keys an answer by the compiler's version, the arguments, the input and the whole source, so a cached answer is the compiler's answer for exactly that program. Two jobs that share the cache at once could write the same answer file together; the runner takes one job at a time unless its `capacity` is raised.
|
||||
- **`CI`.** Each workflow sets `CI=true` (Gitea and GitHub set it too). Under it a test that would skip for a missing tool fails instead, because a skip in CI is how a gap hides: python3 (`tests/mojo.test.mjs`, `tests/dafny.test.mjs`, `tests/library.test.mjs`), the fetched Dafny (`tests/dafny.test.mjs`), a git checkout to read the library pointer from (`tests/chain.test.mjs`), and in pplus-web pandoc for the Quarto tests.
|
||||
- **What the host needs besides pixi:** git, curl, tar, `sha256sum`, bash, and Node for `actions/checkout` itself; a C compiler (`cc`) for the C references `library-check` builds. Without `cc` those checks are recorded as skipped, `library/status.json` then differs from the committed one, and the last step of `test.yml` fails and says so.
|
||||
Registering one, on the machine that will run it (a Gitea act_runner, as its documentation describes: download it, `act_runner generate-config > config.yaml`, run it as a service under a user of its own):
|
||||
|
||||
```bash
|
||||
# in a container, the image the deploy runner uses (anything with git, Node, curl, tar and bash will do):
|
||||
act_runner register --instance https://<your-gitea> --token <runner token> --no-interactive \
|
||||
--labels "heavy:docker://<your-gitea-host>/infra/quarto-deploy:1.9.38"
|
||||
# or on the machine's own shell (host mode), where the caches below last from job to job:
|
||||
act_runner register --instance https://<your-gitea> --token <runner token> --no-interactive --labels "heavy:host"
|
||||
```
|
||||
|
||||
Take the token from the organization's runner settings, so that pplus-core and pplus-library both see the runner. In `config.yaml`, `runner.capacity` is the number of jobs at once: leave it at 1 unless each job has a cache directory of its own (`.tools/`, below). The workflows use no `container:` and no `services:` of their own, and each needs one change to run on GitHub Actions: `runs-on: ubuntu-latest` in place of the label. A pull request from a fork would run its own code on that machine; each job's `if:` runs pull requests only from branches of the repository itself.
|
||||
|
||||
- **pixi.** `PIXI_VERSION` at the top of each workflow is the pixi that reads the lock. The first step uses the runner's pixi when it is that version; otherwise it downloads that release from GitHub into `$PPLUS_CI_CACHE/pixi-<version>/` (default `~/.cache/pplus-ci`), checks its SHA-256 (written in the workflow), and puts it on the PATH. No sudo. A runner that keeps its home between jobs (host mode) fetches it once; a container fetches it in every job. To move pixi: change `PIXI_VERSION` and both hashes in every workflow, in one commit, and check that `pixi install --locked` still accepts the lock.
|
||||
- **The environment.** `pixi install --locked` fails if `pixi.toml` and `pixi.lock` disagree, so the runner never resolves anything new. On Linux the environment has a C compiler (`gcc` from conda-forge, which is also `cc` on the PATH), because `mojo build` links every program with the C compiler it finds and stops with "unable to find suitable c compiler for linking" when there is none, as in a slim container image; `library-check` builds the C references with it and one test a C harness. The C library it links against is pinned too (`sysroot_linux-64` 2.34): Mojo 1.2.0's runtime libraries need glibc 2.34, and against conda-forge's default, 2.28, no Mojo program links. So under `pixi run` it is this compiler that links, on a workstation as on the runner, whatever the system has. Where pixi's package cache lasts, each package is downloaded once; a container downloads the whole environment in every job.
|
||||
- **`.tools/`.** The pinned Dafny (`pixi run dafny-fetch`, about 150 MB) and `mojo-local`'s answers live in `.tools/`; in CI `.tools` is a link to `$PPLUS_CI_CACHE/core-tools`, so where that directory lasts neither is fetched or rebuilt on every push. `mojo-local` keys an answer by the compiler's version, the arguments, the input and the whole source, so a cached answer is the compiler's answer for exactly that program. Two jobs that share the cache at once could write the same answer file together; the runner takes one job at a time unless its `capacity` is raised.
|
||||
- **Dafny without ICU.** The release zip carries its own .NET runtime but not ICU, and .NET aborts without it unless told not to look: in the deploy runner's image, which has none, the fetched Dafny printed no version and aborted. `scripts/dafny.py` (and `scripts/library.mjs` and `tests/dafny.test.mjs`, which run the executable themselves) set `DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1`, and `dafny-fetch` fails if the Dafny it fetched does not start.
|
||||
- **`CI`.** Each workflow sets `CI=true` (Gitea and GitHub set it too). Under it a test that would skip for a missing tool fails instead, because a skip in CI is how a gap hides: python3 (`tests/mojo.test.mjs`, `tests/dafny.test.mjs`, `tests/library.test.mjs`), the fetched Dafny (`tests/dafny.test.mjs`), `cc` (`tests/library-harness.test.mjs`), a git checkout to read the library pointer from (`tests/chain.test.mjs`), and in pplus-web pandoc for the Quarto tests.
|
||||
- **What the runner needs besides pixi:** git, curl, tar, `sha256sum`, bash, and Node for `actions/checkout` itself. Everything else is pinned here: the compilers by `pixi.lock`, Dafny by its hash.
|
||||
- **A slow or busy machine changes how long the gate takes, not what it finds.** The gate first ran on the deploy server, a small virtual machine: the tests that take 20 seconds on a workstation took 15 to 22 minutes there, work that takes 5 milliseconds took more than 3 seconds, and ten tests failed on correct code. Two things in the tests depended on speed then and do not now. A time limit in a test ("this reads in under 2 seconds", there to catch work that grows with the square of its input) is processor time at the machine's pace: `tests/helpers/pace.mjs` times a fixed piece of work and multiplies the limit by how much longer it takes than on the machine the limits were written on, never by less than 1. And the prover's time on the clock, 3 seconds for each routine in the Prove tab, is the caller's: the tests, `pixi run cli prove` and `library-check` give it none, so the counted budgets decide alone and the answer is the same everywhere ([limits.md](limits.md#proving-and-testing)). The fuzzers keep their clock (`--budget-ms`, below): on a slower machine `prove-fuzz` proves less, never more.
|
||||
|
||||
## The fuzzers' seeds
|
||||
|
||||
|
||||
+1
-1
@@ -46,7 +46,7 @@ The reference's "Limits" and "What stops a run" tables (`src/common/reference.ts
|
||||
- **What the parser reads.** A line inside more than 200 blocks, or brackets, calls, unary operators or types nested more than 200 deep, is one `too-deep` error (`MAX_NESTING` in `parse.ts`); so is a chain of more than 400 operands (`1 + 1 + ...`, `MAX_CHAIN`). The two are counted apart and a chain leans one level per operand, so brackets that each hold a long chain would multiply them; an expression whose tree is taller than 400 levels is therefore `too-deep` as well (`MAX_TREE`, measured where each expression is complete). No later stage (checker, analysis, translators) walks a taller tree; `tests/pseudo-robustness.test.mjs` takes the tallest through every one of them.
|
||||
- **What a run holds.** One collection holds at most 2,000,000 values (an ARRAY's declared size, `FILL`, a range walked by `FOR EACH` or listed for a quantifier), and a text at most 10,000,000 characters (`MAX_VALUES`, `MAX_TEXT` in `interp.ts`); past either the run stops (kind `value`) instead of allocating.
|
||||
- **How long a run goes.** The step limit (20,000 in the app's runs, `maxSteps`) stops a loop that may never end (kind `limit`). The positions FORALL, EXISTS, SUM and sets written by rule visit are counted apart, up to 100 per step allowed, so one quantifier inside another cannot run without end. Calls go 400 deep (`maxDepth`, kind `depth`); a recursion whose call sits inside many nested blocks can exhaust the engine underneath before 400 calls (with ten blocks around the call, somewhere between 200 and 300 calls, varying from run to run), which is the same stop, at the line the run was on.
|
||||
- **The Prove tab's budget.** The verifier's work is counted, not timed (`work` in `logic.ts`): one obligation's proof, one subroutine's paths and invariants, its proofs, and the whole program each have a budget, the same on every machine, and past one the answer is open ("too large to prove here") or the subroutine partial, never proved ([limits.md](limits.md#proving-and-testing)). A wall-clock budget stays behind them as a last resort: `provePseudo` gives the verifier 3 seconds per unit, then tries generated inputs on each routine within a counted budget of steps and also for up to 3 seconds (`checkProperties` with `maxWork` and `budgetMs`): a slow routine is tried on fewer inputs, not waited for.
|
||||
- **The Prove tab's budget.** The verifier's work is counted, not timed (`work` in `logic.ts`): one obligation's proof, one subroutine's paths and invariants, its proofs, and the whole program each have a budget, the same on every machine, and past one the answer is open ("too large to prove here") or the subroutine partial, never proved ([limits.md](limits.md#proving-and-testing)). A wall-clock budget stays behind them as a last resort: `provePseudo` gives the verifier 3 seconds per unit, then tries generated inputs on each routine within a counted budget of steps and also for up to 3 seconds (`checkProperties` with `maxWork` and `budgetMs`): a slow routine is tried on fewer inputs, not waited for. The 3 seconds are the Prove tab's (`PROVE_BUDGET_MS`); a caller gives another with `provePseudo(parsed, { budgetMs })` or `Workspace.proofBudgetMs`, and the command line, `library-check` and the tests give none, so their answers do not depend on the machine ([limits.md](limits.md#proving-and-testing)).
|
||||
- **Recording.** A recorded run (the stepper, the table of values) keeps a picture of the variables after every step, and a picture copies each variable the step changed. A large collection that changes at many steps is copied at each of them, so once the copies of a run come to more than 10,000,000 values (about 80 MB of numbers; `maxRecorded` in `interp.ts`), the trace ends with a step of kind `unrecorded` that says so, and the run goes on without recording: its output, result, stop and line counts are the whole run's. No program of the corpus, the exercises or the lessons comes near it (the largest copies about 42,000). `tests/pseudo-robustness.test.mjs` holds the cut with a small budget.
|
||||
|
||||
- **Checker errors and stops added in the 2026-11 audits** (each in `tests/pseudo-robustness.test.mjs`, `tests/numeric.test.mjs` or `tests/reference.test.mjs`): `too-deep` (the limits above), `literal-range` for a whole-number literal past 2^53 − 1 and a FLOAT or FLOAT32 literal past its largest value, `recursive-type`, `array-of-array`, `array-length` (an ARRAY given a LIST or ARRAY of another length, when both are written as numbers), `bad-member` (a SET of anything but single values). Stops: the collection and text caps, the quantifier positions, an ARRAY given a value of another length at a store, a call or a RETURN, a FOR counter declared with a sized type leaving its type, a partial sum or product of `SUM_OF` or `DOT` past 2^53 − 1, `CHR` of a surrogate half, `TO_FLOAT` or `INPUT` of text that is not a decimal number, and NaN or an infinity passed to a FLOAT parameter from outside without `USE IEEE`.
|
||||
|
||||
+1
-1
@@ -73,7 +73,7 @@ The prover's work is counted, not timed: each term it builds or visits, each cas
|
||||
| The routines of one program the fuzzer tests | `FUZZ_ROUTINES`, `src/common/workspace.ts` | the first 100 | A report with no cases: "Not tested: generated inputs cover the first 100 routines of a program, and this is routine N." |
|
||||
| The fuzzer's work for a whole program | `FUZZ_PROGRAM_INPUTS`, `src/common/workspace.ts` | 600 inputs' worth: 1,800,000 at the default budget | A report with no cases: "Not tested: the routines before this one took all the work the tests of one program may take (1,800,000 steps)." |
|
||||
|
||||
The wall-clock budget the Prove tab gives each subroutine (3 seconds) stays behind these as a last resort. Measured on the machine this was written on: 20 nested `ABS` (or a matho pane `f(x) = ||||...x...||||` of 20 bars) took a 1 GB heap down before, and now ends in 0.1 seconds in a 128 MB heap; a subroutine at `UNIT_WORK` takes 0.7 to 2 seconds, one at `PROOF_WORK` about 1 second, and a program at `PROGRAM_WORK` about 4 seconds, each in a 128 MB heap. The most any program of the corpus takes is 234,000 units for one obligation, 145,000 for one subroutine's paths, 783,000 for its proofs and 886,000 for the whole program. `prove-fuzz` makes larger programs than anyone writes: the most in seeds 9137 and 22051 (800 programs each) are 768,000, 35,000,000 (a subroutine its 1.5-second wall-clock budget had already cut short), 5,100,000 and 38,000,000, and the proved obligations of both seeds are the same with the budgets as without. The cross-check is counted the same way: one input of a routine whose ENSURE nests 8 quantifiers visited 5,000,000 positions (7 seconds) before, and the whole cross-check of a routine now takes at most about 2 seconds; the most any routine of the corpus takes is 347,000 (`match-off-by-one`). The fuzzer runs a routine's inputs within its budget in about a second, a program's in about 2 (a program of 2,000 small routines, the first 100 fuzzed: 2 seconds); the most the corpus takes is 400,000 for one routine (kmers' `kmer_counts` in Mojo), 1,300 for one input, and every corpus program's report is the same as before.
|
||||
The wall-clock budget the Prove tab gives each subroutine (3 seconds, `PROVE_BUDGET_MS` in `src/services/pseudo.ts`) stays behind these as a last resort. It is the caller's: `provePseudo` and `Workspace.proofBudgetMs` take another, and what has to say the same thing on every machine takes none (`pixi run cli prove` unless `--budget-ms` gives one, `library-check`, the tests), so the counted budgets decide alone. The clock is the one part of a proof that depends on the machine: on CI's runner, where the README's example took more than its 3 seconds (it takes 5 milliseconds on the machine this was written on), 4 of its 6 obligations stayed open. A slow or busy device shows the same in the Prove tab: fewer obligations proved, more open, none wrongly proved. Measured on the machine this was written on: 20 nested `ABS` (or a matho pane `f(x) = ||||...x...||||` of 20 bars) took a 1 GB heap down before, and now ends in 0.1 seconds in a 128 MB heap; a subroutine at `UNIT_WORK` takes 0.7 to 2 seconds, one at `PROOF_WORK` about 1 second, and a program at `PROGRAM_WORK` about 4 seconds, each in a 128 MB heap. The most any program of the corpus takes is 234,000 units for one obligation, 145,000 for one subroutine's paths, 783,000 for its proofs and 886,000 for the whole program. `prove-fuzz` makes larger programs than anyone writes: the most in seeds 9137 and 22051 (800 programs each) are 768,000, 35,000,000 (a subroutine its 1.5-second wall-clock budget had already cut short), 5,100,000 and 38,000,000, and the proved obligations of both seeds are the same with the budgets as without. The cross-check is counted the same way: one input of a routine whose ENSURE nests 8 quantifiers visited 5,000,000 positions (7 seconds) before, and the whole cross-check of a routine now takes at most about 2 seconds; the most any routine of the corpus takes is 347,000 (`match-off-by-one`). The fuzzer runs a routine's inputs within its budget in about a second, a program's in about 2 (a program of 2,000 small routines, the first 100 fuzzed: 2 seconds); the most the corpus takes is 400,000 for one routine (kmers' `kmer_counts` in Mojo), 1,300 for one input, and every corpus program's report is the same as before.
|
||||
|
||||
## What the compiler does
|
||||
|
||||
|
||||
+1
-1
@@ -88,7 +88,7 @@ With `--statements` it writes whole programs instead (20 to 60 lines, 100 per se
|
||||
|
||||
Two kinds of difference are counted and not reported as failures, because the compiler decides them ([trust.md](trust.md#what-a-translation-says)): numbers that differ only in their last digits (the compiled program's `std.math` is its own; its arithmetic is the pseudo's, since every program is built with contraction off), and, for lines that use `**`, `exp` or `log2`, what those lose in Mojo 1.2.0 (they are good to 9 to 13 digits). What the compiler does wrong on its own is listed in `UPSTREAM` in the fuzzer, with the probe that records it; read that list on every version bump.
|
||||
|
||||
Neither task needs or replaces godbolt. A built program that traps is run with its core-dump limit set to 1, so a deliberate crash does not reach the desktop's crash handler. CI runs both in the pinned environment, which brings the compiler: `pixi run ci` runs `mojo-local` and a short `mojo-fuzz`, and the nightly run fuzzes longer ([ci.md](ci.md)). Run both yourself after changing the translator, the checker or the interpreter, and after a version bump.
|
||||
Neither task needs or replaces godbolt. Both need a C compiler: `mojo build` compiles a program and has the `gcc` or `cc` it finds link it, against a C library no older than glibc 2.34. The pinned environment brings one on Linux (`pixi.toml`), so under `pixi run` the system's is not used; a program that compiles and cannot be linked is reported by `mojo-local` as not linked, the machine's fault and not Mojo's answer, and is not kept in its cache. A built program that traps is run with its core-dump limit set to 1, so a deliberate crash does not reach the desktop's crash handler. CI runs both in the pinned environment, which brings the compiler: `pixi run ci` runs `mojo-local` and a short `mojo-fuzz`, and the nightly run fuzzes longer ([ci.md](ci.md)). Run both yourself after changing the translator, the checker or the interpreter, and after a version bump.
|
||||
|
||||
## Reading the status
|
||||
|
||||
|
||||
Reference in new issue
Block a user