Build, test, deploy / site (push) Successful in 9m10s
The deploy server is a slow container with other work to do. Its job ran core's tests, which failed there (no C compiler, no ICU, limits written for a fast machine), so nothing was published. - pixi run ci, which the deploy job runs, is the site's tests and core's mojo-status-strict at the pinned commit. Core's own tests and everything that builds Mojo are core's gate, on a runner of its own; pixi run core-test stays for moving the pointer. - The workflow and docs/deploy.md describe the runner as it is: a new container for every job, not a host shell. - core moves to b10a5c8: a pinned C compiler for the Mojo compiler to link with, a Dafny that starts without ICU, time limits at the machine's pace, and a prover whose time on the clock is the caller's (the Prove tab keeps its 3 seconds).
57 lines
11 KiB
Markdown
57 lines
11 KiB
Markdown
# Deploying
|
|
|
|
The site builds against pplus-core, the git submodule `core/`, at the commit this repository pins. A checkout needs it (`git clone --recurse-submodules`, or `git submodule update --init`); `npm run build` builds core first. To take a newer core: `git -C core pull` (or check out a tag), `pixi run test`, and commit the new `core` pointer with what it changes on this side.
|
|
|
|
`pixi run site` tests the app and assembles `site/`; `pixi run deploy [slug] [user@host]` does that and publishes it. CI (`.gitea/workflows/deploy.yml`) runs `pixi run ci` (the site's tests and core's `mojo-status-strict`) on every push and pull request, and on a push to `main` checks the chain of pointers (`pixi run release-check`), assembles and publishes.
|
|
|
|
The site is seven things: `index.html`, `styles.css`, `fonts/` (IBM Plex, served by the site so a visit asks no font host for anything; `fonts/README.md` says where the files came from), `styles/` (the course's widget styles, which the Hardware area's machine views use), `dist/`, `core/` (pplus-core's built `core/dist/`, which `index.html`'s import map names as `#core/`), and the Quarto extension (`quarto/_extensions/pplus`, without an engine copy), which the With Quarto page loads (`pplus.css`, `pplus.js`) and links to file by file. `scripts/assemble.sh` copies them into `site/` with a `VERSION` file holding the commit, a `build.json` naming the three commits the site was built from (below), and the license and notices (`LICENSE.md`, `NOTICE.md`, `THIRD_PARTY.md`, which the footer links), then `scripts/assemble-page.mjs` adds two things to `site/index.html`: the commit as `<meta name="pplus-version">`, which the footer shows, and a `<link rel="modulepreload">` for every module the page's imports reach, after the import map, so a first visit asks for all of them at once instead of one level of imports at a time (`tests/load.test.mjs` holds the list to the import graph and the inline scripts to their hashes). And `scripts/deploy.sh <slug> [user@host]` publishes `site/` to `/srv/www/<slug>` as an atomic release:
|
|
|
|
```
|
|
/srv/www/pplus -> symlink to the live release
|
|
/srv/www/pplus.releases/ one directory per deployed commit, the five newest kept
|
|
```
|
|
|
|
The swap is one rename, so nobody sees a half-copied site, and a rollback is `ln -sfn /srv/www/pplus.releases/<sha> /srv/www/pplus`. A pre-existing plain directory at `/srv/www/<slug>` is moved to `<slug>.releases/before-releases` on the first run.
|
|
|
|
**Caddy**: [`Caddyfile`](Caddyfile) is the site block (root on the symlink, `file_server`, a missing path that looks like a page sent to `/` (the app's routes are after the `#`, and the page names its files relative to itself, so it cannot be served in place at a deeper path) while a missing file is a plain 404, and caching: `no-cache` on the page, the modules and the stylesheets, so the browser asks again on every load and gets a bodiless 304 while a file is unchanged; 30 days on the fonts, which change only when `fonts/` is replaced; `no-store` on `VERSION`; nothing is `immutable`, since no file name here changes with its content; the license and notices are served as plain text) and a snippet of security headers it imports: a Content-Security-Policy (everything from the site itself, the two inline scripts of `index.html` by their hashes, `connect-src` open only to godbolt.org for Compile & run, and `frame-ancestors` so course pages can embed views), `Referrer-Policy: no-referrer`, `X-Content-Type-Options: nosniff`, and a `Permissions-Policy` that turns off the camera, microphone, location, payments and USB. `tests/csp.test.mjs` fails when an inline script of `index.html` changes without its hash, or when the page names another origin; a new backend for Compile & run needs its origin in `connect-src`. The live shortcodes and `live="click"` put the app in an iframe, so a Quarto site on another origin needs that origin added to `frame-ancestors`; the rendered views need nothing from the host. Caddy issues the certificate itself; add the block to `/etc/caddy/Caddyfile` and `systemctl reload caddy`. The tests read the file as text; `pixi run caddy-check` serves an assembled `site/` with a real Caddy under it (localhost, plain HTTP) and holds the headers, the caching and both missing-path answers. Run it after changing the Caddyfile.
|
|
|
|
**Gitea Actions** (`.gitea/workflows/deploy.yml`): the deploy job runs on the label `quarto-deploy`. On the droplet's runner that label is a container image (`infra/quarto-deploy` on the same Gitea: Node, git, rsync, curl, Quarto and a pixi) with the droplet's `/srv/www` bound into it. A job gets a new container and runs in it as root; nothing it writes outside `/srv/www` is there for the next job. The image is general on purpose, so everything else the job needs comes from `pixi.lock`:
|
|
|
|
1. `actions/checkout` with `submodules: recursive` (the job's token must be able to read pplus-core and pplus-library while they are private);
|
|
2. **pixi**, at the version `PIXI_VERSION` names in the workflow: the image's when it is that version, else that release from GitHub, checked by SHA-256 (on this runner, every job);
|
|
3. `pixi install --locked`: Node, Python, pandoc and the Mojo compiler, exactly as `pixi.lock` has them. With no cache between jobs the whole environment is downloaded each time;
|
|
4. `pixi run ci`: every test of the site (the Quarto ones run in pandoc and fail, not skip, if it is missing), and core's `mojo-status-strict` at the pinned commit (offline, a second);
|
|
5. on `main`: `pixi run release-check`, `scripts/assemble.sh`, and `scripts/deploy.sh` into `/srv/www`.
|
|
|
|
**The deploy job is light on purpose.** The droplet is a server with other work, and slow: the site's tests take 2.5 minutes there (half a minute on a workstation), and core's tests, which the job ran until November 2026, took 15 to 22. So the job holds the site to its own tests and to core's stored compiler answers, and nothing in it builds a Mojo program. Core's tests, `library-check`, `mojo-local` and the fuzzers are pplus-core's gate (`pixi run ci` there), which runs on a second runner with the label `heavy` (core's `docs/ci.md` has its setup) and on a developer's machine. Before moving the `core` pointer, run `pixi run core-test` here, or let `pixi run release-bump` run core's gate (below): the deploy job will not catch a core that fails its own tests.
|
|
|
|
What the runner has to bring is small: git and Node (for `actions/checkout`), bash, curl, tar, `sha256sum`, and a `/srv/www` the job can write (or rsync and SSH to the machine that has it, below). The same workflow runs on a runner in host mode (a label registered as `quarto-deploy:host`: steps run on the machine's own shell as the runner's user, which then needs those tools and write access to `/srv/www`); there pixi's package cache and the fetched pixi in `~/.cache` last from job to job.
|
|
|
|
`SITE_SLUG` (repository variable) names the directory and defaults to the repository name; set it to `pplus`. `SITE_URL` turns on a smoke test that fetches `/VERSION` from the live site and fails the job if it is not the deployed commit. On GitHub Actions the build and test steps run unchanged with `runs-on: ubuntu-latest`; publishing needs the droplet.
|
|
|
|
**Once the repositories are public**, a pull request from a fork runs its own workflow file and code on this runner, in a container that can write `/srv/www`. Run pull requests only from branches of the repository itself (each workflow's job has an `if:` that does that), or require approval for fork runs in Gitea's repository settings where the version offers it, or give pull requests a separate runner that cannot write the web root.
|
|
|
|
If the runner ever moves off the droplet, `scripts/deploy.sh pplus deploy@droplet` rsyncs over SSH instead; give that runner a key whose public half is in `deploy`'s `authorized_keys`.
|
|
|
|
By hand, on the droplet: `pixi run deploy pplus`. From elsewhere: `pixi run deploy pplus deploy@droplet`.
|
|
|
|
## Moving the pointers
|
|
|
|
The site pins pplus-core by commit (`core/`), and core pins pplus-library (`core/library/`). A change at the bottom reaches the site in three commits, pushed in order: library, then core with its new `library` pointer and what it regenerates from it, then the site with its new `core` pointer. Done by hand the first time, it broke twice: a pointer named a commit from before a move, and a pointer named a commit that existed on one machine only. Two tasks (`scripts/release.mjs`) hold the order:
|
|
|
|
- `pixi run release-check` says whether the chain is consistent, and what to run when it is not: each pointer's commit is in the checkout, the checkout is at it and has no uncommitted change, and the commit is on the submodule's branch (`main`) on its remote. It asks the remote with `git ls-remote` (never a fetch); with `--offline`, or when the remote cannot be asked, it reads the last fetch and says "cannot tell" where that cannot settle it. It also checks that the generated files are current (core's `library.mjs emit --check`, `corpus.py emit` writing nothing new, `quarto-css --check`) and that Mojo is pinned alike in `core/library/mojo/pin.json`, both `pixi.toml` files and both `pixi.lock` files, with Node and Python alike in core and the site. Exit status 1 on a problem.
|
|
- `pixi run release-bump`, after a change committed in `core/library` (or in `core`): moves core's pointer to the library checkout, regenerates core's registry and the corpus's generated files, runs core's gate and `library-check`; then builds the site against that core, regenerates the Quarto extension's files, runs the site's tests; and prints the commits to make and push, in order. With `--commit` it makes the commits; it never pushes, tags or fetches. `--no-gate` skips the gates.
|
|
|
|
`tests/chain.test.mjs` runs the cheap half of `release-check` with every `npm test`: it fails when a pointer names a commit its checkout does not have, or a checkout is behind its pointer, and names the command that fixes it (`git submodule update --init --recursive`). Core's `tests/chain.test.mjs` does the same for its own pointer.
|
|
|
|
## Versions and tags
|
|
|
|
Today core is 0.2.0 and the site 0.1.0 (`package.json` and `pixi.toml` say the same, and `tests/tasks.test.mjs` keeps them so); the library has no version, and nothing is tagged. A built site says what it is made of: `site/build.json` (`scripts/build-info.mjs`) holds each repository's version, commit and `git describe --tags --always --dirty`, and the Mojo the corpus is pinned to. The app does not show it yet; an About line that reads `build.json` is the remaining step.
|
|
|
|
Proposed for the first public release (the owner's to decide; nothing here creates a tag):
|
|
|
|
- Each repository keeps its own version, in its `package.json` and `pixi.toml` (the library, which has neither, in its tag only), and a release is an annotated tag `vX.Y.Z` on the commit the repository above pins.
|
|
- First release: pplus-library `v0.1.0`; pplus-core `v0.3.0`, because its exports changed since 0.2.0 (`linear()`, `LIBRARY_COMMIT`); the site `v0.2.0`. The version bumps are commits of their own, made before the tags.
|
|
- Tag bottom up, after `pixi run release-check` is clean and each commit is pushed: the library at the commit core pins, core at the commit the site pins, then the site. `git describe` in `build.json` then reads `v0.3.0` instead of a hash.
|
|
- `release-bump` stays out of tagging: moving a pointer is not a release. What it could do later is refuse to move a pointer onto a commit that is behind a pinned tag, and `release-check --release` could require each pinned commit to carry a `v*` tag that matches its `package.json`.
|