Skip to content

Benchmarks and suites

Built-in, your own and imported benchmarks (SWE-bench, Aider polyglot), and suites.

benchmarks

Benchmarks and suites: built-in, your own and imported (SWE-bench, Aider polyglot).

Benchmark

Benchmark(data: dict[str, Any] | str)

Bases: Handle

A benchmark: its stages and prompts, starter workspace, grader and reference solution.

prompts property

prompts: list[Any]

The prompt of each stage, in order.

spec

spec() -> dict[str, Any]

The editable definition of one of your benchmarks (what create_benchmark() and update() take).

update

update(spec: dict[str, Any]) -> 'Benchmark'

Replace one of your benchmarks with a new spec (built-in ones are read-only: duplicate() them first).

duplicate

duplicate(new_id: str) -> 'Benchmark'

An editable copy under a new id (3-61 lowercase letters, digits or underscores).

export

export(path: Union[str, Path, None] = None) -> bytes

The benchmark as a zip (its folder: benchmark.json, workspace, grader, reference); written to path if given. import_benchmark() reads it back.

validate

validate(wait: bool = True, quiet: bool = False) -> Union[dict[str, Any], Job]

Check the grader: the starter workspace must fail it and the reference solution must pass. Returns the report (or the Job with wait=False). Needs Docker and the sandbox image.

Suite

Suite(data: dict[str, Any] | str)

Bases: Handle

A named group of benchmarks; pass its id wherever benchmarks are expected.

update

update(*, name: str | None = None, benchmarks: Sequence[Any] | None = None, description: str | None = None) -> 'Suite'

Rename it, change its description or its benchmarks (built-in suites are read-only).

list_benchmarks

list_benchmarks(category: str | None = None, search: str | None = None) -> list[Any]

Benchmarks in the data folder (built-in, your own and imported), as dicts with id, title, category, …

list_suites

list_suites() -> list[Any]

Suites (named groups of benchmarks): built-in ones plus yours and those created by imports.

import_swebench

import_swebench(dataset: str = 'verified_mini', *, limit: int | None = None, instance_ids: Sequence[str] | None = None, quiet: bool = False) -> dict[str, Any]

Import SWE-bench tasks as benchmarks: dataset "verified" (500), "lite" (300) or "verified_mini" (50). Grading needs the official harness: setup(swebench=True). Returns the new benchmark ids and suite id.

import_polyglot

import_polyglot(languages: Sequence[str] | None = None, *, quiet: bool = False) -> dict[str, Any]

Import Aider's polyglot benchmark (cpp, go, java, javascript, python, rust). Needs the toolchain image: setup(polyglot=True).

import_catalog

import_catalog() -> dict[str, Any]

What can be imported and what is ready: SWE-bench datasets and its harness, the polyglot languages, image and exercise cache, and how many of each are imported.

install_swebench_harness

install_swebench_harness(quiet: bool = False) -> dict[str, Any]

Install the official SWE-bench harness into its own virtual environment (about 250 MB).

build_polyglot_image

build_polyglot_image(quiet: bool = False) -> dict[str, Any]

Build the polyglot toolchain image (C++, Go, Java, JavaScript, Rust; a few GB).

remove_imported

remove_imported(kind: str) -> int

Delete every imported benchmark of one kind ("swebench" or "polyglot") and its suites; returns how many benchmarks were removed. They can be imported again at any time.

benchmark

benchmark(benchmark_id: str) -> Benchmark

A benchmark by id, with everything about it.

create_benchmark

create_benchmark(spec: dict[str, Any]) -> Benchmark

Create your own benchmark from a spec, the same one the benchmark editor builds: id, title, prompts per stage, starter files, hidden tests or checks for the grader, an optional reference solution. See docs/BENCHMARKS.md.

import_benchmark

import_benchmark(source: Union[str, Path, bytes]) -> Benchmark

Add a benchmark from a zip (a path or its bytes), such as one Benchmark.export() made on another machine.

suite

suite(suite_id: str) -> Suite

A suite by id.

create_suite

create_suite(name: str, benchmarks: Sequence[Any], description: str = '') -> Suite

A new suite of benchmarks (Benchmark objects or ids).