Task-type scenarios¶

← Back to README

The runner must handle more than one kind of engineering request. This catalogue freezes one scenario per request type, each with an independent oracle that scores the delivered workspace. The oracle never reads the runner's own reports and never uses a candidate's green tests as the sole verdict. Runner honesty scenarios (crash, budget, lying Builder) live separately under test-scenarios/; the complexity ladder they both serve is the progressive testing plan.

ID	Request type	What is asked	What the oracle executes
`BUGFIX-01`	Fix a bug in committed code	Blank or whitespace-only name must exit 2 with usage; existing tests kept; no README or file changes	Five invocations; seeded test names still present; candidate suite passes on the candidate and fails on the seeded bug (the regression is real); README byte-identical; no extra files
`FEATURE-01`	Add a feature to an existing project	`list --tag TAG` case-insensitive filter in a seeded notes CLI with saved data; old commands and store format unchanged	Delivered `notes.json` must exist and be byte-identical to the seed; filters on mixed-case tags against a scratch copy; no-match, empty tag and missing value behavior; scratch store unchanged after reads; `add` still works; seeded tests kept and suite passes
`ARCH-01`	System architecture task	ADR, `architecture/components.json`, per-component contracts, stub packages, and a checker that enforces the declared import boundaries	ADR sections; exact component set, required edges, acyclic, disjoint ownership; contract operations are callable in `services/<id>/api.py`; AST import checks include relative, comma-separated and nested-package imports, excluding docstring examples. The delivered checker must pass the candidate and an allowed-import control, then reject an injected forbidden static import without a traceback or syntax error. Injection preserves docstrings and future imports
`PROGRAM-01`	Large requirement: many components, parallel build, combine, deploy descriptors	Catalog, cart, checkout and gateway as separate stdlib HTTP processes, shared contracts, `run_local.py`, an e2e test, and compose descriptors	Checks all four `contracts/<shape>.json` files for object type and required field declarations. Runs delivered `run_local.py` on free ports; independently drives the gateway journey (list, add two items, checkout total, cart cleared, empty-cart 409, order lookup, unknown order 404); sends SIGTERM and checks launcher exit and closed ports within five seconds. Delivered unittest e2e discovery must run at least one test and exit successfully; assertion quality is not checked. Compose lists all four services; descriptors are never executed
`UI-01`	Design → implementation (Figma stand-in)	Implement a frozen `design/spec.json` as one self-contained `web/index.html` plus a stdlib structural test	Static: ids, labels, `role=status`, viewport meta, DOM order, no external resources, tokens. In Chromium via Playwright: state/disabled maps, transitions, cancelled note, one-row desktop layout, stacked 44px targets at 375px, no horizontal scroll. Without a browser the result is `DEFERRED`, never `PASS`

The pre-existing LIVE-01, LIVE-02, LIVE-05 and LIVE-06 scenarios stay registered unchanged. python3 tools/live_trial.py --list prints the whole registry.

Where the pieces are¶

tools/task_scenarios.py: seeds, briefs, oracles, and for PROGRAM-01 a ready program manifest (program_manifest) so the same scenario can run as one run with parallel milestone Builders or as autocode program.
tools/scenario_references.py: a handwritten reference delivery per scenario. They are the oracle's positive control and a starting point for a scripted fixture provider; they are not model output.
tests/test_scenario_oracles.py: exercises positive and targeted negative controls. The references pass (UI defers without a browser); seed-only controls do not pass; and targeted breakages fail on the expected row (a fix without a regression test, a case-sensitive filter, a filter that rewrites or deletes the store, a vacuous architecture checker, relative and comma import violations, unrelated checker crashes, a wrong checkout total, an uncleared cart even with green no-op candidate tests, missing contracts, empty or miswired launchers, failed shutdown, empty or failing e2e suites, a missing service, a wrong UI transition, an unstacked mobile layout). Positive controls include alternate stdlib JSON serializers.

Running a scenario¶

Offline oracle controls (no provider, no model spend):

python3 -m unittest tests.test_scenario_oracles

Score a workspace you delivered by any route (a manual run, a dashboard run, a program's integration worktree). The report lands in the evidence bundle:

python3 tools/live_trial.py BUGFIX-01 --score-only /path/to/delivered/workspace

Drive a scenario through the real CLI. The driver seeds the workspace, launches autocode, serves the explicit human gates (--answer, --approve-goal) from the saved state, and scores the result with the oracle. The runner's own completion claim is never the verdict:

# Harness check only: the scripted fixture provider implements LIVE-01.
python3 tools/live_trial.py LIVE-01 --profile fixture --workspace /tmp/trial

# Authorized live trial of a task type.
python3 tools/live_trial.py FEATURE-01 --profile glm53-mimo --workspace /tmp/trial \
  --i-authorize-live-model-spend

# The large requirement as a program: each workstream is its own reviewed run.
python3 tools/live_trial.py PROGRAM-01 --mode program --profile glm53-mimo \
  --workspace /tmp/trial --i-authorize-live-model-spend

Exit codes and verdicts follow testing: PASS (0), HONEST_BLOCKER (2, a recorded pause rather than delivered work), FALSE_COMPLETE/FAIL/ERROR (1). --score-only exits 2 for DEFERRED.

--i-authorize-live-model-spend never authorizes deployment. The driver's separate --authorize-deployment flag is required to schedule deployment workstreams; PROGRAM-01 generates descriptors as ordinary code and does not need that flag.

In program mode the first gate is the program agreement (WAITING_AGREEMENT_APPROVAL, see programs). As with plans, only the offline fixture profile approves it; a live profile stops there, an HONEST_BLOCKER, until a person reads it with autocode program show and approves its exact token. The driver also stops at any other program-level pause (PAUSED_INTEGRATION_CHECK, PAUSED_INHERITANCE, ...), which the controller raises again before it advances any workstream, except PAUSED_MERGE_CONFLICT: a conflict holds only its own workstream, so the driver keeps serving the other workstreams' gates.

The scenario harness has a program mode of its own, separate from live_trial.py: a catalog scenario with category = "program" is driven through autocode program plan, derive, show, approve and run passes, with real workstream runs. The program-notes-cli scenario covers the walking skeleton, two parallel workstreams, an accepted interface change request that re-checks the producer and its consumers, and the final check by journey; its broken variants are caught by the cumulative checks (HONEST_BLOCKER) and only by the hidden journey test (FALSE_COMPLETE):

.venv/bin/python scenarios/run.py run program-notes-cli --fake

Unlike live_trial.py, that driver approves the agreement and every workstream's plan under a live profile too, as its single-run driver approves the plans it is shown. See scenarios/README.md, Programs.

When a scenario counts as reliably passing¶

The live qualification rule (#110, as practised): a rung passes only with 3 fresh live PASS runs in a row on one profile, with zero FALSE_COMPLETE, each judged by that scenario's own oracle. Fresh means every run starts from a clean checkout with no reused workspace or run state. A rung passes on any one profile, and the profile that passed is recorded (for example glm53-openai, default, claude-tiers). Fake runs (--fake) are used on every change; live runs are spent only at qualification rungs and need explicit spend authorization.

The FX04 frozen-reference fixture for LIVE-04's 7-state × 3-viewport matrix was not created. That matrix is owned by the Figma design-coverage track (#250 coverage manifests, #251 rendered comparison, #297 deterministic PNG comparison); tests/test_catalogue_t12.py records the remaining gap.

Limits, stated plainly¶

All five task types were attempted on 2026-09-26 with the glm53-mimo profile. None delivered a completed product within the recorded budgets: planning and report repair exhausted the attempts. The registry records ERROR, not PASS; VALIDATION.md distinguishes external interruptions from enforced deadlines, includes independent post-stop scores, and records source/driver identities. These bounded failures are not a ranking of the models or proof that the tasks cannot be solved. Offline controls still cover only references and selected defects. A later BUGFIX-01 retest with stable report-repair inputs reached the Builder but still exhausted its 1,200-second budget without delivery; its baseline remains ERROR.
One case per task type is a capability example, not a general pass rate. Comparing PROGRAM-01 in run and program modes reuses the task and scorer, but program mode additionally supplies a handwritten decomposition and has separate child budgets and approvals. It is a workflow comparison, not an isolated measurement of parallelism.
The scripted fixture provider implements only the greeting scenario, so --profile fixture exercises the harness for LIVE-01 only. Other scenarios need a live profile or --score-only on a delivered workspace.
UI-01 is a stand-in for the Figma route. The real design workflow (autocode ui ... --build, Figma) needs the Codex Figma plugin and has no offline oracle. UI-01 tests the implementation half against a frozen reference; it does not test Figma inspection or design generation.
ARCH-01 examines static Python imports, not dynamic imports or the semantic quality of architecture decisions. Its paired checker controls rule out the demonstrated syntax/import-crash false positives, not every possible unrelated failure or an adversarial checker tailored to the controls.
PROGRAM-01 checks contract field declarations, not complete JSON Schema type semantics. Delivered e2e discovery checks successful, nonempty execution, not assertion quality or candidate-test coverage. Independent HTTP checks decide the sampled behavior even when the candidate tests are green no-ops.
PROGRAM-01 checks compose service names, not image builds, networking or deployability. Its deploy workstream is kind: code because it writes descriptors only; it still forbids running Docker or reaching external systems. Integration depends on that descriptor workstream so final verification includes its files. Nothing is deployed. Its contracts workstream is the walking skeleton: the contracts plus every process answering /health and the catalog listed through the gateway, verified by tests/test_skeleton.py (the manifest's program check) before any other workstream starts. The oracle does not score that file.
These oracles execute candidate Python locally; they are not a security sandbox. Subprocess deadlines and process-group cleanup bound ordinary hangs and clean up ordinary descendants, but do not contain hostile code or children that detach into a different process group. Launcher shutdown checks cover SIGTERM and listener closure, not every Ctrl-C race or possible non-listening background worker.
The driver serves default answers to open questions and approves the displayed plan token. That is test automation, not a person's judgement; do not count a driven run as human acceptance where a criterion requires human review.

See also: Program workstreams · Testing plan · Testing