A proof report named a routine as stopped by the clock whenever the generated inputs that cross-check it were cut short, even when every claim of it was proved and nothing was left undecided. On a loaded machine that is the common case: a container running the site's tests on two processors reported it for the simplest proved example. The prover running out of time still counts always; the inputs' clock counts when a claim of the routine is still open.
This commit is contained in:
5 files changed
+30
-9
No files matched your search
+1
-1
@@ -23,7 +23,7 @@ Mojo 1.2.0, held to the real compiler on the machine as well as on godbolt.org,
|
||||
|
||||
### Added
|
||||
|
||||
- A proof report says when the time on the clock, not the proof, stopped a routine: `ProofReport.clock` has the budget and the routines it stopped, each such unit has `timedOut` and a note, and `checkProperties` answers `timedOut` when the clock cut its inputs short. The site uses it to offer more time.
|
||||
- A proof report says when the time on the clock, not the proof, stopped a routine: `ProofReport.clock` has the budget and the routines it stopped, each such unit has `timedOut` and a note, and `checkProperties` answers `timedOut` when the clock cut its inputs short. Inputs cut short count only where a claim of the routine is still open: on a routine whose every claim is proved nothing was left undecided. The site uses it to offer more time.
|
||||
- `pixi run mojo-local` builds every Mojo program core holds or generates on the pinned local compiler and holds the checker, the interpreter and the translator to it; `pixi run mojo-fuzz` does the same with generated programs.
|
||||
- `pixi run mojo-fuzz --statements` generates whole programs (loops, branches, collections, records, routines with BYREF), builds each with the pinned compiler and holds its output to the pseudo's line by line; `pixi run prove-fuzz` holds the prover to runs of generated programs. `pixi run ci` runs both on a fixed and a moving seed, and `pixi run nightly` runs them longer.
|
||||
- The Mojo interpreter runs what the translator writes: typed list literals, `List[T](slice)`, `ord` and `chr`, std.math's exact functions (`gcd`, `lcm`, `comb`, `hypot`, `fma`, `nextafter`, `ulp`, `copysign`, `isinf` and the others), `atof` of a decimal, and the `match` statement. What it declines on purpose (a fractional power, the transcendental functions, random numbers) says why.
|
||||
|
||||
+1
-1
@@ -73,7 +73,7 @@ The prover's work is counted, not timed: each term it builds or visits, each cas
|
||||
| The routines of one program the fuzzer tests | `FUZZ_ROUTINES`, `src/common/workspace.ts` | the first 100 | A report with no cases: "Not tested: generated inputs cover the first 100 routines of a program, and this is routine N." |
|
||||
| The fuzzer's work for a whole program | `FUZZ_PROGRAM_INPUTS`, `src/common/workspace.ts` | 600 inputs' worth: 1,800,000 at the default budget | A report with no cases: "Not tested: the routines before this one took all the work the tests of one program may take (1,800,000 steps)." |
|
||||
|
||||
The wall-clock budget the Prove tab gives each subroutine (3 seconds, `PROVE_BUDGET_MS` in `src/services/pseudo.ts`) stays behind these as a last resort. It is the caller's: `provePseudo` and `Workspace.proofBudgetMs` take another, and what has to say the same thing on every machine takes none (`pixi run cli prove` unless `--budget-ms` gives one, `library-check`, the tests), so the counted budgets decide alone. The clock is the one part of a proof that depends on the machine: on CI's runner, where the README's example took more than its 3 seconds (it takes 5 milliseconds on the machine this was written on), 4 of its 6 obligations stayed open. A slow or busy device shows the same in the Prove tab: fewer obligations proved, more open, none wrongly proved. The report says when that happened (`clock.stopped` names the routines, each has `timedOut` and a note), and the site lets the reader give the prover more time (its `docs/app.md`). Measured on the machine this was written on: 20 nested `ABS` (or a matho pane `f(x) = ||||...x...||||` of 20 bars) took a 1 GB heap down before, and now ends in 0.1 seconds in a 128 MB heap; a subroutine at `UNIT_WORK` takes 0.7 to 2 seconds, one at `PROOF_WORK` about 1 second, and a program at `PROGRAM_WORK` about 4 seconds, each in a 128 MB heap. The most any program of the corpus takes is 234,000 units for one obligation, 145,000 for one subroutine's paths, 783,000 for its proofs and 886,000 for the whole program. `prove-fuzz` makes larger programs than anyone writes: the most in seeds 9137 and 22051 (800 programs each) are 768,000, 35,000,000 (a subroutine its 1.5-second wall-clock budget had already cut short), 5,100,000 and 38,000,000, and the proved obligations of both seeds are the same with the budgets as without. The cross-check is counted the same way: one input of a routine whose ENSURE nests 8 quantifiers visited 5,000,000 positions (7 seconds) before, and the whole cross-check of a routine now takes at most about 2 seconds; the most any routine of the corpus takes is 347,000 (`match-off-by-one`). The fuzzer runs a routine's inputs within its budget in about a second, a program's in about 2 (a program of 2,000 small routines, the first 100 fuzzed: 2 seconds); the most the corpus takes is 400,000 for one routine (kmers' `kmer_counts` in Mojo), 1,300 for one input, and every corpus program's report is the same as before.
|
||||
The wall-clock budget the Prove tab gives each subroutine (3 seconds, `PROVE_BUDGET_MS` in `src/services/pseudo.ts`) stays behind these as a last resort. It is the caller's: `provePseudo` and `Workspace.proofBudgetMs` take another, and what has to say the same thing on every machine takes none (`pixi run cli prove` unless `--budget-ms` gives one, `library-check`, the tests), so the counted budgets decide alone. The clock is the one part of a proof that depends on the machine: on CI's runner, where the README's example took more than its 3 seconds (it takes 5 milliseconds on the machine this was written on), 4 of its 6 obligations stayed open. A slow or busy device shows the same in the Prove tab: fewer obligations proved, more open, none wrongly proved. The report says when that happened (`clock.stopped` names the routines: those the prover ran out of time on, each with `timedOut` and a note, and those whose generated inputs were cut short while a claim was still open), and the site lets the reader give the prover more time (its `docs/app.md`). Measured on the machine this was written on: 20 nested `ABS` (or a matho pane `f(x) = ||||...x...||||` of 20 bars) took a 1 GB heap down before, and now ends in 0.1 seconds in a 128 MB heap; a subroutine at `UNIT_WORK` takes 0.7 to 2 seconds, one at `PROOF_WORK` about 1 second, and a program at `PROGRAM_WORK` about 4 seconds, each in a 128 MB heap. The most any program of the corpus takes is 234,000 units for one obligation, 145,000 for one subroutine's paths, 783,000 for its proofs and 886,000 for the whole program. `prove-fuzz` makes larger programs than anyone writes: the most in seeds 9137 and 22051 (800 programs each) are 768,000, 35,000,000 (a subroutine its 1.5-second wall-clock budget had already cut short), 5,100,000 and 38,000,000, and the proved obligations of both seeds are the same with the budgets as without. The cross-check is counted the same way: one input of a routine whose ENSURE nests 8 quantifiers visited 5,000,000 positions (7 seconds) before, and the whole cross-check of a routine now takes at most about 2 seconds; the most any routine of the corpus takes is 347,000 (`match-off-by-one`). The fuzzer runs a routine's inputs within its budget in about a second, a program's in about 2 (a program of 2,000 small routines, the first 100 fuzzed: 2 seconds); the most the corpus takes is 400,000 for one routine (kmers' `kmer_counts` in Mojo), 1,300 for one input, and every corpus program's report is the same as before.
|
||||
|
||||
## What the compiler does
|
||||
|
||||
|
||||
+1
-1
@@ -93,7 +93,7 @@ export interface Obligation {
|
||||
}
|
||||
|
||||
export interface ProofReport {
|
||||
/** `timedOut`: the time on the clock ran out on this routine (the prover's paths, or the inputs that cross-check it). */
|
||||
/** `timedOut`: the time on the clock ran out on this routine: on the prover's paths, or on the inputs that cross-check it while a claim of it is still open. */
|
||||
units: Array<{ name: string; kind: string; line: number; obligations: Obligation[]; inferred: string[]; notes: string[]; partial: boolean; timedOut?: boolean }>;
|
||||
counts: { proved: number; broken: number; open: number; total: number };
|
||||
/**
|
||||
|
||||
+15
-6
@@ -125,7 +125,11 @@ export const PROVE_BUDGET_MS = 3000;
|
||||
* the same on every machine (the command line, library-check, a test) passes Infinity: the counted budgets then
|
||||
* decide alone, and they run out at the same point everywhere.
|
||||
*/
|
||||
export interface ProveOptions { budgetMs?: number }
|
||||
export interface ProveOptions {
|
||||
budgetMs?: number;
|
||||
/** The clock of the generated inputs alone, when it is not the prover's (`budgetMs`): the two are separate budgets of the same length. */
|
||||
inputsBudgetMs?: number;
|
||||
}
|
||||
|
||||
export function provePseudo(parsed: Parsed<Study>, options: ProveOptions = {}): ProofReport {
|
||||
const s = parsed.program;
|
||||
@@ -136,8 +140,12 @@ export function provePseudo(parsed: Parsed<Study>, options: ProveOptions = {}):
|
||||
// a run breaks is "broken", the rest stay "open".
|
||||
const brokenKeys = new Set<string>();
|
||||
const brokenAt = new Set<string>();
|
||||
// the routines the clock stopped, the prover on its paths or the inputs before all were tried
|
||||
const stopped = new Set<string>(r.units.filter((u: any) => u.timedOut).map((u: any) => u.name as string));
|
||||
// the routines the clock stopped: the prover on its paths, and the generated inputs before all were tried. The
|
||||
// second counts only where something is left open (below): inputs cut short on a routine whose every claim is
|
||||
// proved left nothing undecided, and on a loaded machine that is the common case.
|
||||
const proverStopped = new Set<string>(r.units.filter((u: any) => u.timedOut).map((u: any) => u.name as string));
|
||||
const inputsStopped = new Set<string>();
|
||||
const inputsBudgetMs = options.inputsBudgetMs ?? budgetMs;
|
||||
// the inputs share a budget of work (steps and quantifier positions), each routine's and the program's: a routine
|
||||
// past it is not tried, and its unproved obligations stay open
|
||||
let left = CROSS_CHECK_PROGRAM;
|
||||
@@ -145,9 +153,9 @@ export function provePseudo(parsed: Parsed<Study>, options: ProveOptions = {}):
|
||||
if (left <= 0) break;
|
||||
try {
|
||||
// and the same wall-clock budget the verifier has, per routine: a slow routine is tried on fewer inputs
|
||||
const pr = checkProperties(s.program, sub.name, { count: 150, budgetMs, maxWork: Math.min(left, CROSS_CHECK_ROUTINE) });
|
||||
const pr = checkProperties(s.program, sub.name, { count: 150, budgetMs: inputsBudgetMs, maxWork: Math.min(left, CROSS_CHECK_ROUTINE) });
|
||||
left -= (pr as { work?: number }).work ?? 0;
|
||||
if ((pr as { timedOut?: boolean }).timedOut) stopped.add(sub.name);
|
||||
if ((pr as { timedOut?: boolean }).timedOut) inputsStopped.add(sub.name);
|
||||
// a failure that says which half of a contract it broke (an INVARIANT before the first pass or after one, a
|
||||
// DECREASES below 0 or not smaller) counts against that obligation only
|
||||
for (const f of pr.failures) { brokenKeys.add(`${f.kind}@${f.line}`); brokenAt.add(`${f.kind}@${f.line}:${f.col}:${f.phase || ''}`); if (!f.phase) for (const ph of ['init', 'keep', 'bound', 'step']) brokenAt.add(`${f.kind}@${f.line}:${f.col}:${ph}`); }
|
||||
@@ -161,7 +169,7 @@ export function provePseudo(parsed: Parsed<Study>, options: ProveOptions = {}):
|
||||
return brokenKeys.has(`${kindKey(o.kind)}@${o.line}`) ? 'broken' : 'open';
|
||||
};
|
||||
const units = r.units.map((u: any) => ({
|
||||
name: u.name, kind: u.kind, line: u.line, partial: !!u.partial, timedOut: stopped.has(u.name),
|
||||
name: u.name, kind: u.kind, line: u.line, partial: !!u.partial, timedOut: proverStopped.has(u.name),
|
||||
inferred: (u.inferred || []).flatMap((x: any) => (typeof x === 'string' ? [x] : (x.facts || []).map((f: string) => `line ${x.line}: ${f}`))),
|
||||
notes: (u.notes || []).map((n: any) => n.text || String(n)),
|
||||
obligations: u.obligations.map((o: any) => {
|
||||
@@ -173,6 +181,7 @@ export function provePseudo(parsed: Parsed<Study>, options: ProveOptions = {}):
|
||||
};
|
||||
}),
|
||||
}));
|
||||
for (const u of units) if (!u.timedOut && inputsStopped.has(u.name) && u.obligations.some((o: any) => o.status === 'open')) u.timedOut = true;
|
||||
const all = units.flatMap((u: any) => u.obligations);
|
||||
return {
|
||||
units,
|
||||
|
||||
@@ -109,6 +109,18 @@ test('the time on the clock is the caller\'s: run out, it leaves a subroutine pa
|
||||
const stopped = cut.units.find((u) => u.name === 'MAX_SUBARRAY');
|
||||
assert.equal(stopped.timedOut, true);
|
||||
assert.equal(stopped.notes.filter((n) => /ran out of time.*open, not disproved/.test(n)).length, 1);
|
||||
// the inputs' clock alone: cut short on a routine whose every claim is proved, nothing was left undecided, and
|
||||
// the report does not call that running out of time (on a loaded machine it happens to correct, proved programs)
|
||||
const inputsCut = provePseudo(studyOf(src), { budgetMs: Infinity, inputsBudgetMs: -1 });
|
||||
assert.deepEqual(inputsCut.counts, { proved: 6, broken: 0, open: 0, total: 6 });
|
||||
assert.deepEqual(inputsCut.clock.stopped, []);
|
||||
// where a claim is open and the inputs that might break it were not tried, it does
|
||||
// (a fourth power is never negative, which is true and past what this prover shows)
|
||||
const open = 'FUNCTION F(X : INTEGER) RETURNS INTEGER\n ENSURE RESULT >= 0\n RETURN X * X * X * X\nENDFUNCTION\n';
|
||||
const undecided = provePseudo(studyOf(open), { budgetMs: Infinity, inputsBudgetMs: -1 });
|
||||
assert.deepEqual(undecided.counts, { proved: 0, broken: 0, open: 1, total: 1 });
|
||||
assert.deepEqual(undecided.clock.stopped, ['F']);
|
||||
assert.deepEqual(provePseudo(studyOf(open), NO_CLOCK).clock.stopped, []);
|
||||
// left out, the clock is the Prove tab's (whether it is reached is the machine's, and not asserted)
|
||||
assert.equal(provePseudo(studyOf(src)).clock.budgetMs, 3000);
|
||||
// a Workspace hands its own on, and proves again when it changes
|
||||
|
||||
Reference in new issue
Block a user