Benchmarks and suites¶
Built-in, your own and imported benchmarks (SWE-bench, Aider polyglot), and suites.
benchmarks ¶
Benchmarks and suites: built-in, your own and imported (SWE-bench, Aider polyglot).
Benchmark ¶
Benchmark(data: dict[str, Any] | str)
Bases: Handle
A benchmark: its stages and prompts, starter workspace, grader and reference solution.
spec ¶
spec() -> dict[str, Any]
The editable definition of one of your benchmarks (what create_benchmark() and update() take).
update ¶
update(spec: dict[str, Any]) -> 'Benchmark'
Replace one of your benchmarks with a new spec (built-in ones are read-only: duplicate() them first).
duplicate ¶
duplicate(new_id: str) -> 'Benchmark'
An editable copy under a new id (3-61 lowercase letters, digits or underscores).
export ¶
export(path: Union[str, Path, None] = None) -> bytes
The benchmark as a zip (its folder: benchmark.json, workspace, grader, reference); written to path if given. import_benchmark() reads it back.
Suite ¶
Suite(data: dict[str, Any] | str)
Bases: Handle
A named group of benchmarks; pass its id wherever benchmarks are expected.
update ¶
update(*, name: str | None = None, benchmarks: Sequence[Any] | None = None, description: str | None = None) -> 'Suite'
Rename it, change its description or its benchmarks (built-in suites are read-only).
list_benchmarks ¶
list_benchmarks(category: str | None = None, search: str | None = None) -> list[Any]
Benchmarks in the data folder (built-in, your own and imported), as dicts with id, title, category, …
list_suites ¶
list_suites() -> list[Any]
Suites (named groups of benchmarks): built-in ones plus yours and those created by imports.
import_swebench ¶
import_swebench(dataset: str = 'verified_mini', *, limit: int | None = None, instance_ids: Sequence[str] | None = None, quiet: bool = False) -> dict[str, Any]
Import SWE-bench tasks as benchmarks: dataset "verified" (500), "lite" (300) or "verified_mini" (50). Grading needs the official harness: setup(swebench=True). Returns the new benchmark ids and suite id.
import_polyglot ¶
import_polyglot(languages: Sequence[str] | None = None, *, quiet: bool = False) -> dict[str, Any]
Import Aider's polyglot benchmark (cpp, go, java, javascript, python, rust). Needs the toolchain image: setup(polyglot=True).
import_catalog ¶
import_catalog() -> dict[str, Any]
What can be imported and what is ready: SWE-bench datasets and its harness, the polyglot languages, image and exercise cache, and how many of each are imported.
install_swebench_harness ¶
install_swebench_harness(quiet: bool = False) -> dict[str, Any]
Install the official SWE-bench harness into its own virtual environment (about 250 MB).
build_polyglot_image ¶
build_polyglot_image(quiet: bool = False) -> dict[str, Any]
Build the polyglot toolchain image (C++, Go, Java, JavaScript, Rust; a few GB).
remove_imported ¶
remove_imported(kind: str) -> int
Delete every imported benchmark of one kind ("swebench" or "polyglot") and its suites; returns how many benchmarks were removed. They can be imported again at any time.
create_benchmark ¶
create_benchmark(spec: dict[str, Any]) -> Benchmark
Create your own benchmark from a spec, the same one the benchmark editor builds: id, title, prompts per stage, starter files, hidden tests or checks for the grader, an optional reference solution. See docs/BENCHMARKS.md.