Skip to content

Harnesses and agents

Built-in and custom harnesses, and your own Python or TypeScript agent as a harness.

harnesses

Agent harnesses: built-in and custom, images, Dockerfiles, previews and portable runtimes.

Harness

Harness(data: dict[str, Any] | str)

Bases: Handle

A coding-agent harness (Pi, OpenCode, Codex CLI, Qwen Code, Crush, Aider, mini-SWE-agent, or your own) and the Docker image it runs in.

status property

status: str

ready (image built), building, failed or missing (fetched fresh).

apis property

apis: list[Any]

API styles it can talk: openai-chat, openai-responses, anthropic.

build

build(wait: bool = True, quiet: bool = False) -> Union['Harness', Job]

Build (or rebuild) its image on top of the sandbox image, then smoke-test it. wait=False returns the Job.

remove_image

remove_image() -> None

Delete its Docker image (build() makes it again).

dockerfile

dockerfile() -> str

The Dockerfile its image is built from.

preview

preview(api_style: str = 'openai-chat') -> dict[str, Any]

Exactly what a run would execute, for a sample endpoint: config files written into the agent's HOME, environment, the first command and the command that continues a multi-stage task.

runtime

runtime(platform: str = '') -> dict[str, Any]

Its portable runtime (used inside benchmarks that bring their own image, such as SWE-bench) for a platform ("linux/amd64", "linux/arm64"; default this machine's): whether it is built, plus its Dockerfile.

build_runtime

build_runtime(platform: str = '', wait: bool = True, quiet: bool = False) -> Union[dict[str, Any], Job]

Build the portable runtime for a platform. Returns its status, or the Job with wait=False.

update

update(manifest: Union[dict[str, Any], str, Path]) -> 'Harness'

Replace a custom harness's manifest (rebuild afterwards for image changes to take effect).

delete

delete() -> None

Remove a custom harness.

list_harnesses

list_harnesses(ready: bool | None = None) -> list[Any]

Built-in and custom harnesses with their image status; ready=True keeps the ones whose image is built.

harness

harness(id_or_name: Union[str, Harness]) -> Harness

A harness by id ("pi", "opencode", …) or name.

add_harness

add_harness(manifest: Union[dict[str, Any], str, Path], build: bool = False, quiet: bool = False) -> Harness

Add your own harness from a manifest (a dict or a JSON file; see harness_schema() and docs/HARNESSES.md): id, name, install lines (Dockerfile instructions), the command with $PROMPT, config files, API styles. build=True builds its image right away.

harness_schema

harness_schema() -> dict[str, Any]

The JSON schema of a custom harness manifest.

agents

Benchmark your own agent: python_harness() turns a Python function into a harness, so an agent built with LangGraph, the OpenAI Agents SDK, smolagents, PydanticAI or plain code runs in the same offline sandbox, behind the same proxy, with the same transcripts, budgets and grading as the built-in harnesses. node_harness() does the same for a JavaScript or TypeScript function: an agent built with the Vercel AI SDK, Mastra, LangChain.js or the OpenAI Agents SDK for JS.

h = llmdyno.python_harness("my_agent:run", requirements=["langgraph==1.0.2", "langchain-openai==1.0.1"])
llmdyno.bench(qwen, "suite_quick", harnesses=h.id)

The Python entry point is called once per stage, in the task's workspace, with any of these keyword arguments it declares: prompt, model, base_url, api_key, workspace, stage, ctx, max_tokens, reasoning. base_url is LLMDyno's OpenAI-compatible proxy (api_key is a placeholder; the proxy adds the real key). It may be async. Whatever it returns is printed. The JavaScript entry point gets one object with the same fields in camelCase (baseUrl, apiKey, maxTokens). Ready-made agents for each framework are in examples/agents/ (Python) and examples/agents/ts/ (TypeScript).

python_harness

python_harness(entrypoint: str, *, requirements: Sequence[str] = (), source: Union[str, Path, None] = None, id: str | None = None, name: str | None = None, version: str = '1', apis: Sequence[str] = ('openai-chat',), multi_stage: str = 'fresh', build: bool = False, quiet: bool = False, replace: bool = True) -> Harness

Make a harness from a Python function ("module:function"). requirements are pip packages installed into the image (pin versions for repeatable results); source is the module's file or package folder (found on your path when omitted). id defaults to "py-". replace=True updates an existing harness with that id. build=True builds the image right away (it also builds on first use).

node_harness

node_harness(entrypoint: str, *, packages: Sequence[str] | None = None, source: Union[str, Path, None] = None, id: str | None = None, name: str | None = None, version: str = '1', apis: Sequence[str] = ('openai-chat',), multi_stage: str = 'fresh', build: bool = False, quiet: bool = False, replace: bool = True) -> Harness

Make a harness from a JavaScript or TypeScript function ("agent.ts:run"; run, or the default export, when no name is given). It runs on Node 24 in the sandbox, which runs TypeScript directly (type annotations only: no enums or namespaces). packages are npm packages installed for it (pin versions, e.g. "ai@7.0.179"); when omitted, the entry file's export const PACKAGES = [...] list is used, and a package.json in the source folder is installed as well. source is the agent's file, or a folder that the entry point's path is relative to (node_modules, .git and build output are left out). The function gets one object: prompt, model, baseUrl, apiKey, workspace, stage, ctx, maxTokens, reasoning. id defaults to "js-".