Skip to content

Plans

Running profiles × harnesses × benchmarks × iterations.

plans

Plans: profiles × harnesses × benchmarks × iterations, checked first, then run in the background.

Plan

Plan(plan_id: str, job_id: str | None = None)

Bases: Handle

A benchmark plan: its runs, live status, and results once it finishes. name, status and runs() are read fresh each time.

status property

status: str

queued, running, paused, done, cancelled or failed.

job property

job: Job | None

The background job running this plan, if it has one.

runs

runs() -> list[Any]

Every run with its status, scores and timing (dicts; run(id) or get_runs() give Run objects).

get_runs

get_runs() -> list[Any]

Every run as a Run object (transcript, files, logs, retry, …).

retry

retry(statuses: Any = ('failed', 'cancelled'), wait: bool = True, quiet: bool = False, on_event: Any = None)

Run the failed and cancelled runs again (or those with the given statuses) and, with wait=True, wait for the plan and return its Results; otherwise return how many runs were queued.

delete

delete(files: bool = True) -> None

Delete the plan, its runs and reviews; files=True also deletes the run folders (workspaces, logs).

wait

wait(quiet: bool = False, on_event: Any = None, transcript: bool = False, progress: bool | None = None) -> 'Results'

Block until the plan stops, printing each run as it finishes, then return its Results. on_event(ev) is called for every live Event (see watch(); transcript=True adds agent events). progress shows a tqdm bar (default: when tqdm is installed and the output is a terminal or notebook). Ctrl+C cancels the plan.

swebench_predictions

swebench_predictions(path: Union[str, Path, None] = None) -> str

The plan's SWE-bench patches in the official predictions JSONL format (written to path if given).

open

open() -> str

Open this plan in the dashboard (starts the web app in the background if it is not running).

live

live()

A Jupyter widget that follows the plan as it runs (progress, finished runs, GPU); needs "llmdyno[notebook]". Display it as the last expression of a cell.

preflight

preflight(models: Any, benchmarks: Union[str, Sequence[str]] = 'suite_quick', harnesses: Union[str, Sequence[str]] = 'pi', *, iterations: int = 1, **options: Any) -> dict[str, Any]

What bench() would check before starting: blocking problems, warnings, and time and cost estimates.

bench

bench(models: Union[Profile, str, Sequence[Union[Profile, str]]], benchmarks: Union[str, Sequence[str]] = 'suite_quick', harnesses: Union[str, Sequence[str]] = 'pi', *, iterations: int = 1, name: str | None = None, budget_usd: float | None = None, quality: bool = True, timeout_multiplier: float | None = None, cloud_concurrency: int | None = None, allow_spill: bool = False, acknowledge_risks: bool = False, wait: bool = True, quiet: bool = False, on_event: Any = None, auto_log: Any = None) -> Union['Results', 'Plan']

Run models × harnesses × benchmarks × iterations and return the scored Results.

models: Profiles (from cloud_model / local_model) or profile names. benchmarks: benchmark or suite ids (see list_benchmarks() and list_suites()); harnesses: pi, opencode, codex, qwen-code, crush, aider, mini-swe-agent. budget_usd caps this plan's cloud spend on top of each provider's own cap. Local models share the GPU one at a time; cloud runs go in parallel. The plan is checked first and PreflightError lists anything that blocks it; danger warnings (spill, out of memory, …) need acknowledge_risks=True. wait=False returns the running Plan instead; on_event(ev) gets every live Event while it runs (see watch()). auto_log sends the results on when the plan finishes: "mlflow", "wandb", "langfuse", "otel", a list of them, or {"mlflow": {"experiment": "x"}, …}.

list_plans

list_plans() -> list[Any]

Every plan in the data folder, newest first.

plan

plan(plan_id: str) -> Plan

A plan by id.