Plans¶
Running profiles × harnesses × benchmarks × iterations.
plans ¶
Plans: profiles × harnesses × benchmarks × iterations, checked first, then run in the background.
Plan ¶
Plan(plan_id: str, job_id: str | None = None)
Bases: Handle
A benchmark plan: its runs, live status, and results once it finishes. name, status and runs() are read fresh each time.
runs ¶
runs() -> list[Any]
Every run with its status, scores and timing (dicts; run(id) or get_runs() give Run objects).
retry ¶
retry(statuses: Any = ('failed', 'cancelled'), wait: bool = True, quiet: bool = False, on_event: Any = None)
Run the failed and cancelled runs again (or those with the given statuses) and, with wait=True, wait for the plan and return its Results; otherwise return how many runs were queued.
delete ¶
delete(files: bool = True) -> None
Delete the plan, its runs and reviews; files=True also deletes the run folders (workspaces, logs).
wait ¶
wait(quiet: bool = False, on_event: Any = None, transcript: bool = False, progress: bool | None = None) -> 'Results'
Block until the plan stops, printing each run as it finishes, then return its Results. on_event(ev) is called for every live Event (see watch(); transcript=True adds agent events). progress shows a tqdm bar (default: when tqdm is installed and the output is a terminal or notebook). Ctrl+C cancels the plan.
swebench_predictions ¶
swebench_predictions(path: Union[str, Path, None] = None) -> str
The plan's SWE-bench patches in the official predictions JSONL format (written to path if given).
open ¶
open() -> str
Open this plan in the dashboard (starts the web app in the background if it is not running).
live ¶
live()
A Jupyter widget that follows the plan as it runs (progress, finished runs, GPU); needs "llmdyno[notebook]". Display it as the last expression of a cell.
preflight ¶
preflight(models: Any, benchmarks: Union[str, Sequence[str]] = 'suite_quick', harnesses: Union[str, Sequence[str]] = 'pi', *, iterations: int = 1, **options: Any) -> dict[str, Any]
What bench() would check before starting: blocking problems, warnings, and time and cost estimates.
bench ¶
bench(models: Union[Profile, str, Sequence[Union[Profile, str]]], benchmarks: Union[str, Sequence[str]] = 'suite_quick', harnesses: Union[str, Sequence[str]] = 'pi', *, iterations: int = 1, name: str | None = None, budget_usd: float | None = None, quality: bool = True, timeout_multiplier: float | None = None, cloud_concurrency: int | None = None, allow_spill: bool = False, acknowledge_risks: bool = False, wait: bool = True, quiet: bool = False, on_event: Any = None, auto_log: Any = None) -> Union['Results', 'Plan']
Run models × harnesses × benchmarks × iterations and return the scored Results.
models: Profiles (from cloud_model / local_model) or profile names. benchmarks: benchmark or suite ids (see list_benchmarks() and list_suites()); harnesses: pi, opencode, codex, qwen-code, crush, aider, mini-swe-agent. budget_usd caps this plan's cloud spend on top of each provider's own cap. Local models share the GPU one at a time; cloud runs go in parallel. The plan is checked first and PreflightError lists anything that blocks it; danger warnings (spill, out of memory, …) need acknowledge_risks=True. wait=False returns the running Plan instead; on_event(ev) gets every live Event while it runs (see watch()). auto_log sends the results on when the plan finishes: "mlflow", "wandb", "langfuse", "otel", a list of them, or {"mlflow": {"experiment": "x"}, …}.