Reorganize paperspec commands and pin dependency versions #2

Merged
alexm merged 25 commits from dev into main 2026-10-15 09:53:06 -04:00
3 changed files with 28 additions and 61 deletions
Showing only changes of commit 2e3917f9b1 - Show all commits

No files matched your search

+19 -46
View File
@@ -1,4 +1,5 @@
# Builds and checks the website on every push to main, and publishes it when SITE_PATH is set; see deploy/README.md.
# Builds and checks the website on every push to main, and publishes it to /srv/www/paperspec; see deploy/README.md.
# The linux-base runner mounts the host's /srv/www into its job containers, so publishing is a local copy.
name: site
on:
push:
@@ -6,14 +7,15 @@ on:
workflow_dispatch:
permissions:
contents: read
# One publish at a time, so two pushes cannot interleave their transfers; a later run waits instead of cancelling
# one halfway through.
concurrency:
group: site-publish
cancel-in-progress: false
defaults:
run:
shell: bash
env:
SITE_ROOT: /srv/www
SITE_SLUG: paperspec
jobs:
site:
runs-on: linux-base
@@ -22,63 +24,34 @@ jobs:
- uses: actions/checkout@v4
with:
persist-credentials: false
# --locked fails when pixi.lock is out of date with pixi.toml instead of solving a new environment.
- name: Restore the environment
uses: actions/cache@v4
with:
path: .pixi/envs
key: pixi-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('pixi.lock') }}
restore-keys: pixi-${{ runner.os }}-${{ runner.arch }}-
- name: Install the environment
run: pixi install --locked
# Everything but the byte-for-byte rebuild, including that every page, link and $id of the site resolves.
- name: Check the repository and the site
run: pixi run --frozen quick
- name: Build the site
run: pixi run --frozen site --out _site
- name: Publish the site
if: github.ref == 'refs/heads/main'
env:
SITE_HOST: ${{ secrets.SITE_HOST }}
SITE_USER: ${{ secrets.SITE_USER }}
SITE_PATH: ${{ secrets.SITE_PATH }}
SITE_SSH_KEY: ${{ secrets.SITE_SSH_KEY }}
SITE_KNOWN_HOSTS: ${{ secrets.SITE_KNOWN_HOSTS }}
run: |
if [ -z "$SITE_PATH" ]; then
echo "SITE_PATH is not set, so the site was built and checked but not published."
if [ ! -d "$SITE_ROOT" ]; then
echo "$SITE_ROOT is not mounted on this runner, so the site was built and checked but not published."
exit 0
fi
# rsync --delete empties the folder it is given, so take only an absolute path at least two levels deep.
dest=${SITE_PATH%/}
case "$dest" in
/*/*) ;;
*) echo "SITE_PATH must be an absolute folder such as /srv/paperspec, not '$SITE_PATH'." >&2; exit 1 ;;
esac
case "$dest/" in
*/../*|*/./*|*//*|*[[:space:]]*) echo "SITE_PATH must be a plain path, not '$SITE_PATH'." >&2; exit 1 ;;
case "$SITE_SLUG" in
''|*/*|.*) echo "SITE_SLUG must be a plain folder name, not '$SITE_SLUG'." >&2; exit 1 ;;
esac
if [ ! -f _site/index.html ] || [ ! -f _site/404.html ]; then
echo "_site/ has no index.html or 404.html, so it was not published." >&2
exit 1
fi
command -v rsync >/dev/null || { echo "rsync is not installed on the runner." >&2; exit 1; }
# --checksum without --times leaves unchanged files untouched, so their Last-Modified and ETag stay valid
# in caches; --delay-updates and --delete-delay apply the changes together at the end of the transfer.
opts=(--recursive --links --perms --chmod=D755,F644 --checksum --delay-updates --delete-delay --stats)
if [ -z "$SITE_HOST" ]; then
# The runner is the web server.
if [ ! -d "$dest" ]; then
echo "$dest does not exist on the runner." >&2
exit 1
fi
rsync "${opts[@]}" _site/ "$dest/"
exit 0
fi
if [ -z "$SITE_USER" ] || [ -z "$SITE_SSH_KEY" ] || [ -z "$SITE_KNOWN_HOSTS" ]; then
echo "Publishing over SSH needs SITE_USER, SITE_SSH_KEY and SITE_KNOWN_HOSTS as well as SITE_HOST." >&2
exit 1
fi
# The key and host keys live in a private folder that is removed however the step ends.
keys=$(mktemp -d)
trap 'rm -rf "$keys"' EXIT
printf '%s\n' "$SITE_SSH_KEY" > "$keys/id"
printf '%s\n' "$SITE_KNOWN_HOSTS" > "$keys/known_hosts"
chmod 600 "$keys/id"
ssh="ssh -i $keys/id -o IdentitiesOnly=yes -o BatchMode=yes -o StrictHostKeyChecking=yes"
ssh="$ssh -o UserKnownHostsFile=$keys/known_hosts -o ConnectTimeout=20 -o ServerAliveInterval=15"
rsync "${opts[@]}" --compress -e "$ssh" _site/ "$SITE_USER@$SITE_HOST:$dest/"
# in caches; --delay-updates and --delete-delay apply the changes together at the end of the copy, and
# --chmod keeps every file readable by the web server whatever the container's umask.
rsync --recursive --links --perms --chmod=D755,F644 --checksum --delay-updates --delete-delay --stats \
_site/ "$SITE_ROOT/$SITE_SLUG/"
+2 -2
View File
@@ -1,8 +1,8 @@
# Serves the paperspec website built by `python3 tools/site.py` (the `_site/` folder), copied to /srv/paperspec.
# Serves the paperspec website built by `python3 tools/site.py` (the `_site/` folder), copied to /srv/www/paperspec.
# The site mirrors the repository, so every schema is served at its $id and the JSON-LD context at the address
# layers name; this file sets the media types those addresses need and lets any page fetch them.
paperspec.scient.ing {
root * /srv/paperspec
root * /srv/www/paperspec
encode zstd gzip
# Every file may be fetched from any origin: JSON-LD processors and schema validators in browsers need it.
+7 -13
View File
@@ -35,28 +35,22 @@ pixi run python -m http.server --directory _site 8000
## Serve it with Caddy
[Caddyfile](Caddyfile) serves `/srv/paperspec` at `paperspec.scient.ing`.
[Caddyfile](Caddyfile) serves `/srv/www/paperspec` at `paperspec.scient.ing`.
It sets the media types the data needs (`application/ld+json` for the context, `application/schema+json` for schemas, `application/jsonl` for vectors and package files), allows every file to be fetched from any origin, and serves `404.html` for missing paths.
Add it to the server's Caddy configuration, for example with an `import` line, and reload Caddy.
## Publish from CI
The workflow `.gitea/workflows/site.yml` runs on every push to `main`.
It installs the environment from `pixi.lock`, runs `pixi run quick`, builds the site, and publishes it with `rsync` when the repository secret `SITE_PATH` is set.
It installs the environment from `pixi.lock`, runs `pixi run quick`, builds the site, and copies it with `rsync` to `/srv/www/paperspec`, the folder Caddy serves.
A failed check stops the publish, so the site never serves a release that does not pass its own checks.
Without `SITE_PATH` the workflow still builds and checks the site, and says that it did not publish.
| Secret | Holds |
| --- | --- |
| `SITE_PATH` | The folder Caddy serves, such as `/srv/paperspec`: an absolute path at least two levels deep. |
| `SITE_HOST` | The server's host name. Leave it unset when the runner is the web server, and the site is copied to `SITE_PATH` on the runner. |
| `SITE_USER` | The account that owns the site folder. |
| `SITE_SSH_KEY` | A private SSH key whose public half is in that account's `authorized_keys`, and that can do nothing else. |
| `SITE_KNOWN_HOSTS` | The server's host keys, as lines of `known_hosts`. |
The copy is local: the `linux-base` runner starts each job in a container with the host's `/srv/www` mounted at the same path, so no key or secret is involved.
On a runner without that mount the workflow still builds and checks the site, and says that it did not publish.
`SITE_ROOT` and `SITE_SLUG` at the top of the workflow name the folder.
The last three are needed only with `SITE_HOST`.
Take `SITE_KNOWN_HOSTS` from `ssh-keyscan -H <host>` on a network you trust, and compare its fingerprints with `ssh-keygen -lf` on the server; the workflow refuses a server whose key does not match.
Copying only changed files keeps the `Last-Modified` and `ETag` of every other file, so caches stay valid across releases.
Only changed files are copied, so every other file keeps its `Last-Modified` and `ETag` and caches stay valid across releases.
The environment in `.pixi/envs` is cached between runs under the hash of `pixi.lock`, so a fresh container installs it only when the lock file changes.
## Distributing packages