Baseline: tree as shared
This commit is contained in:
commit
659c03bbf8
1471 files changed
+990535
No files matched your search
@@ -0,0 +1,49 @@
|
||||
# The model machine against compiled Mojo
|
||||
|
||||
"Open in Hardware" puts a function from the real compiler's listing on the model machine (`src/engines/asm86`), translated from godbolt's AT&T syntax by `src/mojo/att.ts`. The machine implements a teaching subset of x86-64, so the question is how much of what the Mojo compiler actually emits it can take. This is the census: every function in every listing the corpus holds (`mojo/golden/*.json` answered with `asm`), sent exactly as the button sends it, and what the assembler said. `node scripts/asm-census.mjs` reruns it (add `--ops` for every mnemonic); `tests/asm-coverage.test.mjs` holds the numbers as a ratchet, so a new corpus item that brings a new instruction fails the build rather than a student's session.
|
||||
|
||||
## Where it stands
|
||||
|
||||
| | before | now |
|
||||
|---|---|---|
|
||||
| distinct instructions in the listings | 97 | 97 |
|
||||
| of which outside the model's set | 38 | 5 (all 256-bit AVX2: `vextracti128`, `vinserti128`, `vpblendvb`, `vperm2i128`, `vpalignr`) |
|
||||
| the user's own functions that assemble | 3 of 7 (`main` never did) | 7 of 7 |
|
||||
| standard-library and runtime functions that assemble | — | 438 of 467 |
|
||||
|
||||
`main()` now steps 25–29 instructions and stops at the call into `print` with "stopped: std_io_io_print: standard library, not on the model machine", which is the honest answer: the model has no standard library, and says exactly where the real program would enter it.
|
||||
|
||||
## What was wrong, and what changed
|
||||
|
||||
**The translator dropped size suffixes only for the mnemonics it knew.** AT&T writes `andnq`, `mulxq`, `shlxq`, `xaddq`, `cmpxchgq`, `btsq`, `bswapq`; the Intel pane reads `andn`, `mulx`, … The BMI and atomic mnemonics were missing from the suffix table, so the machine was asked for an instruction called `andnq`, which exists nowhere. They are in the table now, and `lock` (a prefix) is passed through as a prefix rather than treated as a mnemonic with an operand called `xaddq`.
|
||||
|
||||
**The machine lacked the instructions the optimiser likes.** Added, each with a one-sentence gloss for the datapath view and the micro-op model told what it reads and writes:
|
||||
|
||||
- BMI1/BMI2: `andn`, `mulx`, `shlx`, `shrx`, `sarx`, `bzhi`, `blsr`, `blsi`, `blsmsk`, `rorx`, `bextr` (`pdep`/`pext` are named as not modelled)
|
||||
- the atomics behind reference counts: `xadd`, `cmpxchg`, and the `lock` prefix, accepted on the read-modify-writes the ISA allows it on
|
||||
- bit tests that write: `bts`, `btr`, `btc`; `bswap`; `cwd`, `cwde`, `cbw`
|
||||
- `ud2`: the trap the compiler plants on paths that must never run; reaching it stops the run with "the processor raised an invalid-opcode fault"
|
||||
- the 128-bit AVX spellings of what the subset had: `vmovdqu`, `vmovdqa`, `vmovups`, `vmovaps`, `vmovd`, `vmovq`, `vpxor`, `vxorps`, `vpor`, `vpand`, `vpandn`, `vpaddb`, `vpsubb`, `vpcmpeqb`, `vpmovmskb`, `vzeroupper`; three-operand form (`vop dst, a, b`), sources untouched
|
||||
- lanes: `vpinsr{b,w,d,q}`, `vpextr{b,w,d,q}`, `vpshufb`, `vpshufd`, `vpbroadcast{b,d}`, `vpmin/max{ub,ud}`, `vpcmpeq{w,d,q}`, `vpadd{w,d}`, `vpsubd`, `vpsubusb`, `vpsll/vpsrl{w,d,q}`
|
||||
|
||||
**A call into the standard library refused the whole listing.** The assembler resolves every label up front, so one `call std::io::print` meant the function could not be stepped at all, even to the instruction before the call. The translator now appends a stub for every such symbol (`label:` then `ud2 ; …: standard library, not on the model machine`), so the function assembles, steps up to the call, and the stop says what it called.
|
||||
|
||||
**Read-only strings were unreachable.** `lea rax, [rip + static_string_…]` named a label that was data, not code, and the machine has no data directives. The translator now finds the `.asciz` behind each such label, places the bytes in memory (`memory 0x6000 "…"`, octal and hex escapes decoded, from 0x6000 up), writes the address in the instruction's place, and hands the lines to the setup block, where the student can see them.
|
||||
|
||||
## Scalar doubles, parity, shld (added)
|
||||
|
||||
- `addsd subsd mulsd divsd sqrtsd movsd cvtsi2sd ucomisd comisd` and their `v` forms: IEEE binary64, round to nearest even, default NaN `0xFFF8000000000000`, a NaN operand is quieted and the first operand wins. MXCSR exception flags are not modelled.
|
||||
- `ucomisd`/`comisd` set ZF/PF/CF as hardware does (unordered sets all three; OF, SF clear).
|
||||
- PF is a real flag now: set by logic, add/sub and shifts, cleared by `popcnt`, untouched by rotates. `jp jnp jpe jpo`, `cmovp…` and `setp…` follow.
|
||||
- `shld`/`shrd` on 32- and 64-bit operands; count masked to 5/6 bits; count 0 changes nothing, flags included.
|
||||
- Latencies: add/mul 4, div 14, sqrt 18 (unpipelined on port 0), convert 5, compare 3.
|
||||
|
||||
## What stays outside, and why
|
||||
|
||||
- **256-bit AVX2 (`ymm`)**: the model keeps sixteen 128-bit `xmm` registers and no `ymm` halves; the five instructions above exist only in that width. A `ymm` operand is refused with that sentence. Modelling it means widening every vector register and every lane helper; worth doing only when a user function needs it, and none in the corpus does.
|
||||
- **Symbols the listing addresses but does not define**: `KGEN_…` runtime globals, `…@GOTPCREL` function pointers, a `.zero` constant. The assembler names the symbol in its refusal.
|
||||
- **Constant-pool blocks with a symbol in them** (jump tables): not placed; the instruction that loads one is refused by name. Numeric `.LCPI…` blocks (`.quad`, `.long`, `.double`, `.zero`, …) *are* placed, 16-byte aligned from 0x6000, when the code addresses them, and `.LCPI0_0+8(%rip)` works.
|
||||
|
||||
## Keeping it honest
|
||||
|
||||
The census runs against whatever the corpus holds. Seven goldens carry listings today (the `asm`/`both` probes); `pixi run mojo-verify` with `mode: asm` probes adds more, and each addition either assembles or fails `tests/asm-coverage.test.mjs` with the instruction named. The ratchet numbers in that test are the ones in the table above; raise them when the machine learns more, never lower them.
|
||||
@@ -0,0 +1,31 @@
|
||||
# The vendored course code
|
||||
|
||||
Programming+ runs the course site's own language engines and draws the hardware views with the course's own stylesheets, so a program behaves the same in the app as on a course page.
|
||||
|
||||
| Here | From the course repository | What it is |
|
||||
|---|---|---|
|
||||
| `src/engines/pseudo/` | `static/js/pseudo/` | The pseudo language: parser, checker, interpreter, the English (lingo) reader, the prover's verification conditions, translation |
|
||||
| `src/engines/asm86/` | `static/js/asm86/` | The x86-64 model machine, its caches, pipeline and out-of-order core, and the widgets that draw them |
|
||||
| `src/engines/shared/` | `static/js/shared/` | DOM helpers, URL state, text distance, charts |
|
||||
| `styles/course/` | `static/css/base/`, `static/css/widgets/` | The widget stylesheets ([styles/course/README.md](../styles/course/README.md)) |
|
||||
|
||||
The engines are wrapped behind one `LanguageService` interface (`src/core/types.ts`, adapters in `src/services/`). Modules the app does not use are excluded from the build in `tsconfig.json`. Do not edit `src/engines/` casually: a change there is a change to the course's language.
|
||||
|
||||
## How the copies differ from the course
|
||||
|
||||
`pixi run engines-diff <course>/static` compares every file, ignoring whitespace. As of the last check the stylesheets are identical; most engine modules differ only in formatting (they were reindented); and fourteen differ in content, every one of them a change made here and not yet sent back:
|
||||
|
||||
- **pseudo:** the text built-ins `SPLIT`, `JOIN`, `INDEX_OF`, `REPLACE` and the numerical built-ins (`FLOOR`, `CEIL`, `LOG10`, `SIN`, `COS`, `TAN`, `ATAN2`, `PI`, the statistics) in `lang`, `parse`, `check`, `interp`, `translate`, `vcgen`; `PARALLEL FOR` with its race detector; an indexed read `L[I]` recorded as a read of one position, not all of `L` (`interp`); one `reserved-name` error for a keyword used as a name (`parse`); `english.js` exporting `COMPARISONS` and `IN_PHRASES` so the reference can read them; `analyze.js` and `coach.js` treating a variable passed `BYREF` to a procedure that writes it first as an out-parameter, not a read; and `coach.js` no longer warning "growing bound" on a queue drained by `WHILE LENGTH(Q) > 0`.
|
||||
- **asm86:** what compiled Mojo uses beyond the course's subset (`interp.js`, about 630 lines): the BMI bit instructions, atomic read-modify-writes with `lock`, bit tests that write, `bswap`, `ud2`, the 128-bit AVX spellings, scalar double precision, and the double-width shifts; the parity flag (`interp`, `uops`, `views`); the floating-point units and their latencies (`uops`); a small change in `sandbox.js`; and in `hwviews.js` the whole core's arrows (`machineView`): only a lit edge carries an arrowhead, in its own color and on the end its data reached, each view has its own marker ids, and the result's edge into the registers starts at the port that produced it.
|
||||
|
||||
One addition to the pseudo language lives outside the engine, in app code: `DECLARE <name> : <type> <- <value>`, a declaration with its starting value on one line (Mojo's `var total: Int = 0`). `src/services/pseudo-study.ts` reads it by doing what the engine's `study()` does (parse, check, analyze, coach) with one step between parse and check: the `<- value` is blanked out of the line, columns kept, so the engine parses a plain DECLARE, and the value is parsed on its own and put in as an ordinary assignment on the same line. Every app reader of pseudo (`studyOf`, and the lingo, matho and Mojo translators) goes through it; pseudo-to-mojo writes the pair as one `var` line. The two-line form reads as before. If the course takes the form into its parser, this module can go.
|
||||
|
||||
Deciding what happens to these (send them to the course, or make this repository the engines' home and have the course copy from here) is open; until then, `engines-diff` is how to see the gap.
|
||||
|
||||
## Updating from the course
|
||||
|
||||
1. `pixi run engines-diff <course>/static` and read the list.
|
||||
2. For a file that differs only in whitespace, or that changed only on the course side, copy the course's file over the one here.
|
||||
3. For a file that differs in content on both sides, merge by hand, keeping the changes listed above.
|
||||
4. `pixi run test`. The engines have no tests of their own in the course repository; the tests here (`tests/services.test.mjs`, `tests/corpus.test.mjs`, the curriculum and reference tests, `tests/asm-coverage.test.mjs`) are what catch a regression.
|
||||
5. For the stylesheets, follow [styles/course/README.md](../styles/course/README.md).
|
||||
@@ -0,0 +1,38 @@
|
||||
# Adding an exercise
|
||||
|
||||
An exercise is a catalog problem: one file in `src/exercises/`, listed in `src/exercises/index.ts`. The home page, the problem menu, share links, embeds, and the tests pick it up from there. `pixi run test` checks every exercise: its pseudo parses, its call runs, its answer matches `expect`, and its Mojo program (if it names one) exists and prints what it should.
|
||||
|
||||
There are two ways to start one.
|
||||
|
||||
## An exercise around a Mojo program
|
||||
|
||||
```bash
|
||||
pixi run new-exercise rolling-sum # mojo/local/rolling-sum.mojo, src/exercises/rolling-sum.ts, registered
|
||||
pixi run play rolling-sum # after each edit: run, verify on godbolt, build, list the assembly, test
|
||||
```
|
||||
|
||||
For a title or a home-page section other than the defaults, run the script directly: `node scripts/new-exercise.mjs rolling-sum --title "Rolling sum" --group "Loop forms"`. If `mojo/local/rolling-sum.mojo` already exists, it is kept as written and only the exercise is made.
|
||||
|
||||
The exercise names its program (`mojoProgram: 'rolling-sum'`) instead of carrying the source. The Mojo pane shows the file without its header comments, and badges it "verified on Mojo 1.1.0" only when godbolt's answer for that exact source passed the file's own `# expect-stdout:` lines. Editing the file changes its key, so it reads as unverified until `play` (or `mojo-verify`) asks again. The header keys are in [mojo-corpus.md](mojo-corpus.md#mojo-of-our-own).
|
||||
|
||||
With `# mode: both` and `# args: -O0 --debug-level=line-tables` in the header, godbolt also returns the listing, and `play` turns it into the exercise's x86-64 pane (see "The assembly" in [mojo-corpus.md](mojo-corpus.md#the-assembly)).
|
||||
|
||||
## An exercise in lingo, pseudo, matho or speco
|
||||
|
||||
Copy `src/exercises/_template.ts` to `<id>.ts`, fill it in, import it in `src/exercises/index.ts`, and add it to the list (before `blank`, the demo playground, which stays last). Then `pixi run test`.
|
||||
|
||||
## The fields
|
||||
|
||||
| Field | Means |
|
||||
|---|---|
|
||||
| `id` | The URL (`#/<area>/p/<id>`): lowercase words joined by hyphens |
|
||||
| `title`, `group`, `blurb` | The name, the home-page section it files under (a new name makes a new section), and one or two sentences on what the student does |
|
||||
| `meta` | Optional: `lecture`, `level` (`intro`, `core`, `advanced`), `tags`, `concepts`, `minutes`, `page` (where it is taught) |
|
||||
| `sources` | Any of `lingo`, `pseudo`, `matho`, `speco`, `asm`. Pseudo is derived from lingo (or matho, written in LaTeX) when absent, and speco from the pseudo's inline contracts |
|
||||
| `mojoProgram` | The name of a program in `mojo/local/`, shown in the Mojo pane |
|
||||
| `call`, `expect` | The call to run, and what it should return; the tests compare them |
|
||||
| `setup`, `machine` | For assembly: the setup block (registers and memory before the run) and the machine model preset |
|
||||
| `lang` | The pane that opens first |
|
||||
| `area` | The area it opens in: `bridge`, `dev` or `hw`. Absent: Hardware for `asm-*` ids, Developer for a problem that is only a Mojo program, Bridge otherwise |
|
||||
|
||||
`src/exercises/types.ts` documents each field in full.
|
||||
@@ -0,0 +1,141 @@
|
||||
# The Mojo corpus
|
||||
|
||||
Programming+ teaches what Mojo does when it runs, so everything it says about Mojo has to be true of a real compiler at a known version. The corpus is how: material harvested from Modular's own repositories at pinned commits, plus small programs of our own, every one of them checked on the real compiler through Compiler Explorer (godbolt). The pipeline is Python, standard library only, in `scripts/mojo/` (`corpus.py` is its command line); Node is needed for one processor (Mojo Quest keeps its tickets in TypeScript).
|
||||
|
||||
Every command below is a pixi task; the [README](../README.md#commands) lists them all.
|
||||
|
||||
## Mojo of our own
|
||||
|
||||
There are three kinds of Mojo program we write, and each has a folder, a task to start one, and a job:
|
||||
|
||||
| Kind | Folder | Start one | Its job | Shipped in the app |
|
||||
|---|---|---|---|---|
|
||||
| probe | `mojo/probes/` | `pixi run new-probe <name> "<question>"` | Ask the pinned compiler one question (what does `//` do on a negative Int?). Godbolt's answer is the point; it is recorded, not judged. | no, but the checker and interpreter are held to the answers |
|
||||
| exercise program | `mojo/local/` | `pixi run new-exercise <id>` | The Mojo of a catalog problem. It must pass its own `# expect-stdout:` lines. | yes, in `src/mojo/generated/programs.ts`, badged verified when it passed |
|
||||
| stress program | `mojo/stress/` | `pixi run new-stress <name> "<what it exercises>"` | Realistic Mojo that stresses the interpreter. Its `# expect-stdout:` lines are the interpreter's prediction; godbolt confirms or refutes it. | no |
|
||||
|
||||
The loop is the same for all three:
|
||||
|
||||
```bash
|
||||
pixi run new-probe int-floordiv-negative "does // floor toward negative infinity on Int?"
|
||||
# edit mojo/probes/int-floordiv-negative.mojo
|
||||
pixi run mojo-verify-item 'probes/int-floordiv-negative' # ask godbolt (network)
|
||||
pixi run test # the checker and interpreter scoreboards read the new answer
|
||||
```
|
||||
|
||||
For an exercise program, `pixi run play <id>` does the whole loop in one pass (below).
|
||||
|
||||
### The header
|
||||
|
||||
A program starts with comment lines that `corpus.py verify` reads. Only these keys are header lines; the program starts at the first other line, and the Mojo pane hides the header.
|
||||
|
||||
```mojo
|
||||
# probe: what does // do on negative Int? the question, or what the program shows (one line)
|
||||
# stdin: 3\n4\n what input() reads, with \n escapes
|
||||
# mode: run run | asm | both (asm and both also return the listing)
|
||||
# args: -O0 compiler arguments
|
||||
# expect-stdout: -4 a line the program must print (repeatable); implies expect: pass
|
||||
# expect: compile-error pass | fail | compile-error | observe (the default without expect-stdout)
|
||||
# origin: where the program came from free text, for people
|
||||
# status: a note on its state free text, for people
|
||||
```
|
||||
|
||||
**A stored answer belongs to the exact text.** Godbolt's answer is keyed by the compiler, mode, arguments, stdin and the whole source, header included. Any edit, even to a comment, makes the program unverified until it is asked again. That is deliberate: a stale answer is never shown against new code.
|
||||
|
||||
### The fast loop for an exercise program
|
||||
|
||||
```bash
|
||||
pixi run play rolling-sum # one pass
|
||||
pixi run play-watch rolling-sum # again on every save (tests off)
|
||||
```
|
||||
|
||||
One pass is: run the file on the local `mojo` (pixi provides it); `corpus.py verify --item local/<name>` (godbolt, network); `corpus.py status` (offline, rewrites `programs.ts`); `tsc`; `scripts/mojo/listings.mjs`; the tests. A step that cannot run (no local compiler, godbolt unreachable) is reported as skipped and the pass goes on, so offline you still get the build and the tests, with the program shown as unverified rather than guessed at. `node scripts/play.mjs <name> --no-verify` or `--no-test` skips a step.
|
||||
|
||||
### The assembly
|
||||
|
||||
A program whose header says `# mode: both` (or `asm`) and `# args: -O0 --debug-level=line-tables` makes godbolt return the compiler's listing with its answer. `scripts/mojo/listings.mjs` picks the first `def` that is not `main`, translates it and the user functions it calls to the x86-64 pane's syntax exactly as the **Send to x86-64 pane** button does (`src/mojo/att.ts`, `simulatorSetup` for the argument registers), tags each instruction `; mojo line N` with the line of the program the pane shows, and writes `src/mojo/generated/listings.ts`. An exercise that names the program in `mojoProgram` and leaves `sources.asm` empty gets that listing and its setup. No answer for the exact source, mode and arguments means no listing; a stale answer is never reused, because the key includes the source. Without `-O0` the optimizer may inline every function and the listing holds none of yours; the script says so.
|
||||
|
||||
Header lines (`# probe:`, `# mode:`, …) are for `verify`; only those keys are stripped from what the Mojo pane shows. The first line after the header block is the program.
|
||||
|
||||
## Reading the status
|
||||
|
||||
`pixi run mojo-status` judges every item from the stored answers, offline, and writes `mojo/golden/STATUS.md` (a table per corpus, every failure with its reason, and every probe's question and verdict) and `status.json` (`git diff mojo/golden/status.json` shows exactly what changed).
|
||||
|
||||
Each item says what the pinned compiler should do with it: `pass` (compiles, runs, exits 0, and prints what it should or contains the idiom it should), `fail` (that check must not hold: a starter), `compile-error` (must be rejected, with each expected error at its line), `observe` (a probe: the answer is the point), or `skip` (with a reason: newer syntax than the pin, no `main`, not reproducible on public godbolt). Verdicts are `pass`, `fail`, `observed`, `skipped`, and `unverified` (no golden yet).
|
||||
|
||||
The dialect markers in the pin are a cheap first look: `newer` patterns are syntax that arrived after the pinned version (a manual sample using one is skipped), `older` patterns are syntax the pinned version rejects. They only set expectations; verify decides.
|
||||
|
||||
**What has to pass.** `pixi run mojo-status-strict` (and CI) fails only when a program we wrote fails: a probe, an exercise program, or a stress program. Harvested items can fail for reasons that are facts about upstream (a manual sample written for a newer compiler, a test file with no `main`); they are listed in `STATUS.md` to read, not to fix here. `corpus.py status --strict --strict-corpus all` holds every corpus to it.
|
||||
|
||||
## What lives where
|
||||
|
||||
| Path | What it is | Who writes it |
|
||||
|---|---|---|
|
||||
| `mojo/pin.json` | The Mojo version, the godbolt compiler id, each source repository and the ref to use, and the dialect markers | You |
|
||||
| `mojo/mojo.lock.json` | What each ref resolved to: commit, zip sha256, and the directory inside the zip | `fetch`/`harvest` |
|
||||
| `mojo/corpus/` | Processed material, one directory per processor, each item with its provenance | `harvest` |
|
||||
| `mojo/corpus/SOURCES.md` | Which commit and license everything came from, what each processor found, every warning | `harvest` |
|
||||
| `mojo/probes/` | Our questions about the pinned compiler's behavior, one small program each | You (`new-probe`) |
|
||||
| `mojo/local/` | Exercise programs, shipped in the app | You (`new-exercise`) |
|
||||
| `mojo/stress/` | Realistic programs that test the interpreter, not shipped | You (`new-stress`) |
|
||||
| `mojo/golden/<key>.json` | Godbolt's answer for one program, keyed by compiler, mode, arguments, stdin and source | `verify` |
|
||||
| `mojo/golden/STATUS.md` | Every item's verdict against the goldens | `status` (also `harvest`, `verify`) |
|
||||
| `src/mojo/generated/` | What the app reads at run time: `pin.ts` (version, compiler), `tokens.ts` (the words, removed words and the compiler's messages for them), `editor.ts` (comment and indentation rules), `programs.ts` (`mojo/local/` programs with their verdicts), `stdlib.ts` and `types.ts` (which package exports each name, which types are copyable), and `listings.ts` (exercise listings, written by `scripts/mojo/listings.mjs` during `play`) | `harvest`/`emit`; `verify` and `status` refresh `programs.ts` |
|
||||
| `quarto/_extensions/pplus/syntax/mojo.xml` | The same words as a KDE syntax for Pandoc listings | `harvest`/`emit` |
|
||||
|
||||
Commit all of it. The stored answers are what make `pixi run test` and `mojo-status` work offline.
|
||||
|
||||
## Commands in detail
|
||||
|
||||
The tasks wrap `corpus.py`, which takes more options than the tasks pass:
|
||||
|
||||
- `harvest --local NAME=ZIP` uses a zip on disk for one source. `--local-dir DIR` matches every zip in a directory to a source by its marker files, but only for sources whose ref is still `UNRESOLVED`; add `--offline` to let zips stand in for pinned refs too (the lock then says `standsInFor`, and the next online run fetches the real ref).
|
||||
- `harvest --processors quest,tokens` reruns some processors; `--keep` or `--workdir DIR` keeps the extracted sources for inspection.
|
||||
- `verify --item 'probes/life-*' --force` asks again about some items (`pixi run mojo-verify-item` does this); `--dry-run` shows the request and the count and sends nothing; `--corpus diagnostics` adds the compiler's own tests (opt-in: about 140 files); `--limit N` asks about at most N.
|
||||
- `status --strict` exits 1 if a program we wrote fails; `--strict-corpus` picks other corpora.
|
||||
- `GITHUB_TOKEN` (any token with public read access) avoids GitHub's anonymous rate limit when resolving refs.
|
||||
|
||||
## The processors
|
||||
|
||||
| Processor | From | Produces |
|
||||
|---|---|---|
|
||||
| `quest` | mojo-quest | Each ticket's solution (must pass Mojo Quest's own check) and starter (must not) |
|
||||
| `manual` | modular | The manual's tested code samples (must run and exit 0) and every `mojo` block that is a whole program followed by an `output` block (must print it) |
|
||||
| `diagnostics` | modular | The compiler's `expected-error/warning/note` tests, each annotation resolved to its line and tagged with the pass that reports it (`parser`, `lifetimes`) |
|
||||
| `tokens` | modular | Keywords (`TokenKinds.def`), soft keywords and removed words (the parser), decorators (the reference), prelude names (the stdlib) |
|
||||
| `messages` | modular | The parser's and borrow checker's diagnostic wording and fix-its, read from their C++ |
|
||||
| `release` | modular | The pinned release notes, section by section, and whether the snapshot is the pin, newer, or older |
|
||||
| `syntax` | mojo-syntax | The TextMate grammar's tests as plain highlighting cases |
|
||||
| `editor` | vscode-mojo | Comments, brackets, auto-closing pairs and indentation rules |
|
||||
| `skill` | skills | Modular's table of removed syntax and its replacements |
|
||||
| `stdindex` | modular | Which std package exports each public name (`sqrt` in `std.math`), for the compiler's import hint; a name exported from two places is left out; and which prelude functions raise (`input`, `atol`) |
|
||||
| `stdtypes` | modular | Each std struct's declared traits (so which are implicitly copyable) and each trait's default methods (Copyable's `copy`, Equatable's `__eq__`), for the type checks |
|
||||
|
||||
A processor whose input files are missing at the pinned ref is skipped with a warning, and its previous output is kept.
|
||||
|
||||
## Moving to a new Mojo version
|
||||
|
||||
1. Find the release. Mojo releases are tagged `mojo/vX.Y.Z` in `modular/modular` (the platform's own tags are `modular/vX.Y`). Find godbolt's compiler id for it, `mojo_X_Y_Z`, in `https://godbolt.org/api/compilers/mojo`.
|
||||
2. Edit `mojo/pin.json`: `mojo.version`, `godbolt.compiler`, and `sources.modular.ref`. Move the other refs to the commits that match: Mojo Quest's commit whose `src/lib/compile.ts` names the new compiler id; the vscode-mojo, mojo-syntax and skills commits released alongside it.
|
||||
3. Review the dialect markers against the new release notes. Anything in the notes' **Removed** or **Language changes** sections belongs in `older`; anything in `newer` that this release now includes comes out. (`newer` matters only when harvesting a snapshot past a release.)
|
||||
4. `pixi run mojo-harvest`, then read `mojo/corpus/SOURCES.md`: every warning is something to look at, and a `release` line saying `snapshot newer` means the ref is not the release tag.
|
||||
5. `pixi run mojo-verify`. A new compiler id changes every key, so every item is asked again: about 440 requests, two at a time. Run it once, not in a loop; the public server is shared.
|
||||
6. `pixi run mojo-status`, then read `mojo/golden/STATUS.md`. `git diff mojo/golden/status.json` shows exactly which verdicts and probe answers changed between versions. A changed probe answer is a change Programming+'s Mojo engine has to make; a failing Mojo Quest solution or manual sample usually means a language change the markers should know about.
|
||||
7. `pixi run test`, then commit the pin, the lock, the corpus, the goldens, and `src/mojo/generated/` together.
|
||||
|
||||
## Licenses and provenance
|
||||
|
||||
Every item records its repository, commit (or local zip hash), path, and license; `SOURCES.md` summarizes them. Modular's repository files carry Apache-2.0 WITH LLVM-exception headers. Mojo Quest's files carry the same headers, but the repository has no LICENSE file, so its programs are used as test fixtures here; confirm with Modular before showing them to students as exercises. mojo-syntax is MIT, vscode-mojo is Apache-2.0. Manual prose is not copied into the app; hover text and quick fixes will be written in our own words and link to mojolang.org.
|
||||
|
||||
## When something goes wrong
|
||||
|
||||
| Message | Meaning |
|
||||
|---|---|
|
||||
| `a network proxy refused api.github.com` | No route to GitHub from here. Use `--local-dir DIR --offline`, or run elsewhere. |
|
||||
| `GitHub refused the request (403)` | Usually the anonymous rate limit. Set `GITHUB_TOKEN`. |
|
||||
| `GitHub has no ref ...` | The tag or commit in the pin does not exist. |
|
||||
| `the lock was made from a local zip` | The pin has not moved since a local harvest. Pass the same zip with `--local`, or set a ref. |
|
||||
| `GitHub's zip ... hashes to ...` | The archive for a locked commit changed. Rare; accept it with `--allow-hash-change` after a look. |
|
||||
| `no directory in the zip holds all of [...]` | The repository's layout changed. Update that source's `markers` in the pin. |
|
||||
| `<processor>: skipped, ... has no ...` | That input is missing at this ref; the previous output is kept. |
|
||||
| `the quest processor needs Node 22.6+` | Node strips the TypeScript types from Mojo Quest's ticket file. |
|
||||
@@ -0,0 +1,115 @@
|
||||
# Translation coverage
|
||||
|
||||
<!-- Written by tests/sync-coverage.test.mjs. Do not edit; rerun `npm test` to refresh. -->
|
||||
|
||||
Each translator in `src/sync/` is run over every exercise with pseudo (17), every program in `tests/corpus/` (48), and 17 one-construct fixtures (`tests/helpers/sync-programs.mjs`). 3 of the 82 have pseudo that does not read, by design (they teach the notation's error messages), and are left out below. That leaves **79 programs**.
|
||||
|
||||
| From pseudo to | Says it completely | Says it with a gap or a checker error | Cannot say it |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| Mojo | 77 | 2 | 0 |
|
||||
| lingo | 31 | 0 | 48 |
|
||||
| matho | 3 | 0 | 76 |
|
||||
|
||||
Mojo is a program-for-program translation that always produces something: a line with no faithful spelling becomes a visible `raise Error("not translated: pseudo line N")` and a warning. Lingo and matho must read back as the same program, so what they cannot say exactly they refuse, with the construct and its line, and the pane keeps its last good text.
|
||||
|
||||
## Mojo: lines that become a visible gap
|
||||
|
||||
These programs translate, with the listed lines left as a `raise Error`.
|
||||
|
||||
Nothing.
|
||||
|
||||
## Mojo: programs the translator refuses
|
||||
|
||||
Pseudo the course itself rejects is not translated.
|
||||
|
||||
Nothing.
|
||||
|
||||
## lingo: what English has no sentence for
|
||||
|
||||
English is read by `readSteps`, whose grammar has no sentence for these.
|
||||
|
||||
| What it has no form for | Programs | Which |
|
||||
| --- | ---: | --- |
|
||||
| steps outside a routine (every English step belongs to a routine) | 46 | 01-arithmetic.pseudo, 02-temperature.pseudo, 03-compound-interest.pseudo, 04-gcd-recursion.pseudo, and 42 more |
|
||||
| a fixed-length ARRAY | 4 | ex:edit-distance, 05-primes-sieve.pseudo, 11-matrix-multiply.pseudo, 15-game-of-life.pseudo |
|
||||
| a BYREF parameter | 3 | 06-fibonacci-memo.pseudo, 10-insertion-sort.pseudo, 26-swap-byref.pseudo |
|
||||
| a HASHMAP whose values are not counts | 3 | ex:translate-codons, 13-stack-brackets.pseudo, 39-constants-codons.pseudo |
|
||||
| a change to this part of a value | 2 | ex:edit-distance, 16-running-stats.pseudo |
|
||||
| a DEFINE set | 2 | ex:translate-codons, 21-sets.pseudo |
|
||||
| a loop step that is not a whole number written out | 2 | 05-primes-sieve.pseudo, 39-constants-codons.pseudo |
|
||||
| a record | 2 | 12-records.pseudo, 16-running-stats.pseudo |
|
||||
| a record type | 2 | 12-records.pseudo, 16-running-stats.pseudo |
|
||||
| REMOVE(...) | 2 | 13-stack-brackets.pseudo, 14-queue-bfs.pseudo |
|
||||
| the built-in ROWS | 2 | 11-matrix-multiply.pseudo, 15-game-of-life.pseudo |
|
||||
| this kind of position | 2 | ex:edit-distance, 15-game-of-life.pseudo |
|
||||
| a CONSTANT | 1 | 39-constants-codons.pseudo |
|
||||
| a Field value | 1 | 12-records.pseudo |
|
||||
| PARALLEL FOR | 1 | 43-parallel-map.pseudo |
|
||||
| REPEAT ... UNTIL, which English reads as a loop that tests first | 1 | 46-projectile.pseudo |
|
||||
| the built-in CHOOSE | 1 | 47-binomial-and-ph.pseudo |
|
||||
| the built-in CHR | 1 | 08-caesar.pseudo |
|
||||
| the built-in COS | 1 | 46-projectile.pseudo |
|
||||
| the built-in FILL | 1 | 43-parallel-map.pseudo |
|
||||
| the built-in LOG10 | 1 | 47-binomial-and-ph.pseudo |
|
||||
| the built-in SIN | 1 | 46-projectile.pseudo |
|
||||
| the built-in SPLIT | 1 | 40-text-builtins.pseudo |
|
||||
| the built-in TO_INTEGER | 1 | 22-binary-to-decimal.pseudo |
|
||||
| the built-in TO_STRING | 1 | 22-binary-to-decimal.pseudo |
|
||||
|
||||
## matho: what is not an equation
|
||||
|
||||
An equation lowers to one `FUNCTION` that returns a value; a program with output, text, a loop that does more than add up, or steps outside a function is not one.
|
||||
|
||||
| What it has no form for | Programs | Which |
|
||||
| --- | ---: | --- |
|
||||
| steps outside a function (an equation is a function) | 42 | 01-arithmetic.pseudo, 02-temperature.pseudo, 03-compound-interest.pseudo, 04-gcd-recursion.pseudo, and 38 more |
|
||||
| a function that answers LIST (equations answer numbers) | 5 | ex:match, ex:match-proved, ex:match-off-by-one, ex:match-reset, and 1 more |
|
||||
| a function that answers STRING (equations answer numbers) | 5 | ex:revcomp, fixture:string-build, fixture:slice, fixture:case, and 1 more |
|
||||
| a function that answers BOOL (equations answer numbers) | 3 | ex:binary-search, fixture:early-return, fixture:forall |
|
||||
| a loop that is not adding up or multiplying one value (FOR) | 3 | ex:best-position, fixture:loop-down, fixture:break-continue |
|
||||
| the input T, which is not a number or a list of numbers | 3 | ex:first-unknown, ex:gc-fraction, fixture:set |
|
||||
| a DEFINE set | 2 | ex:translate-codons, 21-sets.pseudo |
|
||||
| a function that answers HASHMAP (equations answer numbers) | 2 | ex:tally, ex:kmers |
|
||||
| a record type | 2 | 12-records.pseudo, 16-running-stats.pseudo |
|
||||
| the input A, which is not a number or a list of numbers | 2 | ex:hamming, ex:edit-distance |
|
||||
| the input B, which is not a number or a list of numbers | 2 | ex:count-base, ex:count-base-proved |
|
||||
| a CONSTANT | 1 | 39-constants-codons.pseudo |
|
||||
| a loop with a STEP | 1 | fixture:loop-step |
|
||||
| a position that is not a loop counter plus or minus a number | 1 | fixture:min-max |
|
||||
| a PROCEDURE | 1 | fixture:procedure |
|
||||
| a WHILE loop | 1 | fixture:div-mod |
|
||||
|
||||
## matho: the equation sets
|
||||
|
||||
Each set in `tests/matho.test.mjs` is lowered to pseudo by the matho reader, written back as equations by `pseudoToMatho`, and checked: 26 of 28 round-trip.
|
||||
|
||||
| Equation set | Round-trips | Why not |
|
||||
| --- | :---: | --- |
|
||||
| mean-and-variance | yes | |
|
||||
| mean | yes | |
|
||||
| sum-from-one | yes | |
|
||||
| sum-no-bounds | yes | |
|
||||
| binomial | yes | |
|
||||
| height | yes | |
|
||||
| kinetic | yes | |
|
||||
| gravity | yes | |
|
||||
| omega | yes | |
|
||||
| normal-density | yes | |
|
||||
| ph | yes | |
|
||||
| log2 | yes | |
|
||||
| ln-and-log | yes | |
|
||||
| factorial | yes | |
|
||||
| choose | no | the loop counter K2 (matho writes it as a single letter) |
|
||||
| cosine | yes | |
|
||||
| distance | yes | |
|
||||
| matrix-row | no | a two-dimensional position that is not two loop counters |
|
||||
| nested-sums | yes | |
|
||||
| relu | yes | |
|
||||
| piecewise | yes | |
|
||||
| calls-an-earlier-equation | yes | |
|
||||
| sum-over-a-set | yes | |
|
||||
| floor | yes | |
|
||||
| ceiling | yes | |
|
||||
| absolute | yes | |
|
||||
| max-min | yes | |
|
||||
| mod | yes | |
|
||||
Reference in new issue
Block a user