Skip to content

Using LLMDyno from Python

Everything the app does is also a Python API. import llmdyno, describe the models you want to compare, call bench(), and you get scored results back. It is the same engine, data folder and database as the web app, so a plan you start from a script shows up in the dashboard, and profiles you made in the UI can be used from code.

import llmdyno

gpt = llmdyno.cloud_model("openai", "gpt-6-luna", reasoning_effort="low", budget_usd=2)
qwen = llmdyno.local_model("unsloth/Qwen3.8-27B-GGUF", quant="UD-Q3_K_XL", ctx=49152)

results = llmdyno.bench([gpt, qwen], benchmarks="suite_quick", harnesses="pi")
print(results)

A real run, one local model on the smoke-test task (RTX 5080):

qwen = llmdyno.profile("Qwen3.8 27B Q3_K_XL 48k")
results = llmdyno.bench(qwen, benchmarks="t0_smoke_cli", harnesses="pi")
print(results)
Plan: 1 run(s), about 1 min
  done      Smoke test: word-frequency CLI · Qwen3.8 27B Q3_K_XL 48k · Pi: correctness 1.00 in 0.5 min · $0.0003
Plan done: {'done': 1}
#  Profile                  Harness  Final  Correct  Quality  Runs  Time     Cost
-  -----------------------  -------  -----  -------  -------  ----  -------  -------
1  Qwen3.8 27B Q3_K_XL 48k  Pi       95.5   100.0    81.8     1     0.5 min  $0.0003

The cost of a local run is the GPU's electricity at your rate (Settings); cloud runs are priced per token.

Install

pip install llmdyno                      # add [pandas] for Results.to_pandas(): pip install "llmdyno[pandas]"

Requirements are the same as for the app: Python 3.11+, Docker running (agents work in sandboxes), and for local models a GPU that llama.cpp supports. import llmdyno itself is instant and starts nothing; the first call opens the data folder.

First-time setup

import llmdyno

print(llmdyno.doctor())          # what is installed and what is missing (same checks as the Setup page)
llmdyno.setup(harnesses=["pi", "opencode"])   # sandbox + harness images, llama.cpp for this machine

setup() only does what is missing. It downloads several gigabytes the first time (Docker images, a llama.cpp build). Add swebench=True for the official SWE-bench harness and polyglot=True for the Aider polyglot toolchain image.

Models

A profile is a model plus the exact settings it is benchmarked with. Both constructors are idempotent: calling them again with the same name returns the same profile (updated with any new settings) instead of creating a copy.

Cloud

claude = llmdyno.cloud_model("anthropic", "claude-sonnet-5-5", reasoning_effort="low")
grok = llmdyno.cloud_model("xai", "grok-5", api_key="...", budget_usd=1.50)
ollama = llmdyno.cloud_model("custom", "qwen3-coder:30b", base_url="http://host.docker.internal:11434/v1")
  • Vendors: openai, anthropic, google, deepseek, moonshot, zai, qwen, minimax, xai, mistral, openrouter, groq, together, fireworks, cerebras, or custom for any OpenAI-compatible URL (Ollama, LM Studio, vLLM, a gateway).
  • Keys: api_key=, or the usual environment variable (OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, DEEPSEEK_API_KEY, XAI_API_KEY, OPENROUTER_API_KEY, …), or a key you saved in the app. Keys are stored encrypted in the data folder and only ever reach LLMDyno's per-run proxy, never an agent sandbox.
  • Spend: budget_usd is a hard cap for that provider (new providers default to $5). bench(budget_usd=…) adds a cap for one plan. The proxy refuses requests once a cap is reached.
  • Prices: pricing="auto" (the default) looks the model up in OpenRouter's public catalog; pass {"input": 3.0, "output": 15.0, "cached": 0.3} (USD per million tokens) to set them yourself.
  • Settings: ctx, max_output_tokens, reasoning_effort (minimal, low, medium, high, max), temperature, top_p, api_style (openai-chat, openai-responses, anthropic), extra_body.

Local (llama.cpp)

# from Hugging Face: downloads once (resumable, checksum-verified), then reuses the file
qwen = llmdyno.local_model("unsloth/Qwen3.8-27B-GGUF", quant="UD-Q3_K_XL", ctx=49152, kv_cache="q8_0")

# a file you already have: registered where it is, nothing is copied
gemma = llmdyno.local_model(path="D:/models/gemma-4-12b-Q8_0.gguf", ctx=65536)

print(qwen.check())      # memory fit: verdict (fits, tight, spill, oom, cpu), projected VRAM, warnings, fixes
qwen.test()              # 20-second load + speed + tool-calling test
qwen.update(ctx=32768)   # change a setting

Settings: ctx, cache_k / cache_v (or kv_cache for both), gpu_layers, n_cpu_moe, flash_attn, thinking, max_output_tokens, reasoning_budget, sampling (temperature, top_p, top_k, min_p, …) and the rest of the profile editor's fields. Unknown names raise TypeError rather than being ignored. local_model() warns when the profile is projected not to fit in VRAM.

Existing profiles

llmdyno.list_profiles()                 # everything, or list_profiles("local") / list_profiles("cloud")
qwen = llmdyno.profile("Qwen3.8 27B Q3_K_XL 48k")   # by name (case-insensitive) or id

qwen.update(name="Qwen 27B 48k", notes="baseline", tags=["round2"], ctx=49152)
fast = qwen.duplicate("Qwen 27B 32k", ctx=32768)    # a copy with changed settings
qwen.export("qwen.profile.json")                    # share it; never contains keys
llmdyno.import_profile("qwen.profile.json")         # on another machine (the model must be downloaded there)
llmdyno.profile_defaults("local", model="Qwen3.8-27B-UD-Q3_K_XL")   # what a new profile starts from

Finding and downloading models

llmdyno.search_models("qwen coder")                 # Hugging Face GGUF repositories
info = llmdyno.repo_info("unsloth/Qwen3.8-27B-GGUF")   # quants, sizes, a fit hint for your GPU, license, gated
fit = llmdyno.estimate("unsloth/Qwen3.8-27B-GGUF", "UD-Q3_K_XL", ctx=65536, kv_cache="q8_0")
fit["check"]["verdict"], fit["contexts"]            # will it fit, before downloading (reads the GGUF header)
m = llmdyno.download_model("unsloth/Qwen3.8-27B-GGUF", "UD-Q3_K_XL")   # or wait=False for the Job
llmdyno.recommended_models()

llmdyno.list_models(status="ready")
m = llmdyno.model("Qwen3.8-27B-UD-Q3_K_XL")          # by id, name or "repo:quant"
m.contexts()                                        # memory at each context size on this machine
m.profile(ctx=32768)                                # a profile from it
m.pause(); m.resume(); m.delete(files=True)
llmdyno.import_models("D:/models")                  # register every GGUF in a folder, no copy

Providers

cloud_model() creates providers for you; these manage them directly.

p = llmdyno.provider("anthropic")                   # by id, name, or vendor when you have one of it
p.test()                                            # checks the key and lists models: {"ok", "models", "latency_ms"}
p.models()
p.spent_usd, p.budget_usd
p.update(budget_usd=10); p.reset_spend()
p.set_key("sk-..."); p.remove_key(); p.delete()
llmdyno.add_provider("custom", base_url="http://localhost:8000/v1", name="vLLM box")
llmdyno.list_providers(); llmdyno.vendors()
llmdyno.lookup_price("anthropic", "claude-sonnet-5-5")   # per 1M tokens, from OpenRouter's catalog

Benchmarks

llmdyno.list_benchmarks()                       # built-in, your own and imported tasks
llmdyno.list_benchmarks(category="SWE-bench")
llmdyno.list_suites()                           # suite_quick, suite_round1, imported suites, your own

llmdyno.import_swebench("verified_mini")        # 50 SWE-bench Verified tasks (or "lite", "verified", limit=…)
llmdyno.import_polyglot(["python", "rust"])     # Aider polyglot exercises
llmdyno.import_catalog()                        # what can be imported and what is ready
llmdyno.install_swebench_harness(); llmdyno.build_polyglot_image(); llmdyno.remove_imported("polyglot")

Your own benchmarks use the same spec as the benchmark editor (see BENCHMARKS.md):

b = llmdyno.benchmark("t0_smoke_cli")           # everything about one benchmark: prompts, stages, grader
mine = b.duplicate("my_smoke")                  # built-in ones are read-only; copies are yours
spec = mine.spec(); spec["stages"][0]["prompt"] += " Use type hints."
mine.update(spec)
mine.validate()                                 # starter must fail, reference must pass (needs Docker)
mine.export("my_smoke.zip"); llmdyno.import_benchmark("my_smoke.zip")
llmdyno.create_benchmark({...})                 # from scratch

s = llmdyno.create_suite("My suite", ["my_smoke", "t5_bugfix_semver"])
s.update(benchmarks=["my_smoke"]); s.delete()

Harnesses

llmdyno.list_harnesses(ready=True)              # images built
pi = llmdyno.harness("pi")
pi.build(); pi.remove_image()
pi.dockerfile(); pi.preview("anthropic")        # exactly what a run executes
pi.runtime("linux/arm64"); pi.build_runtime("linux/arm64")   # portable runtime for SWE-bench-style images
llmdyno.add_harness("my-agent.json", build=True)   # your own agent (see HARNESSES.md); .update(), .delete()
llmdyno.harness_schema()

Running

results = llmdyno.bench(
    [gpt, qwen],                          # profiles, or profile names
    benchmarks=["suite_quick", "swe_django__django_11099"],   # benchmark and suite ids
    harnesses=["pi", "opencode"],
    iterations=2,
    budget_usd=1.0,                       # cap for this plan's cloud spend
)

bench() checks the plan first (Docker, images, keys, memory fit, budgets) and raises PreflightError listing what blocks it, before anything starts. Warnings rated danger (VRAM spill, out of memory) need acknowledge_risks=True. llmdyno.preflight(...) returns the same report without running anything.

While it runs it prints each run as it finishes. Ctrl+C cancels the plan cleanly (finished runs are kept). With wait=False you get a Plan back immediately:

plan = llmdyno.bench([gpt, qwen], "suite_round1", wait=False)
plan.status            # queued, running, paused, done, …
plan.runs()            # every run with its status and scores
plan.pause(); plan.resume(); plan.cancel()
results = plan.wait()
plan.open()            # the live plan page in the dashboard
plan.retry()           # run the failed and cancelled runs again
plan.rename("Round 2"); plan.delete(files=True)

Runs

run = plan.get_runs()[0]                  # or llmdyno.run(run_id); Results.runs() rows carry run ids
t = run.transcript()
t.turns()                # one entry per model call: messages sent, reply blocks, finish, tokens, timings, cost
t.tool_calls()           # every tool call with its arguments and what the tool returned
t.thinking(); t.final_text(); t.usage(); t.to_pandas()
run.files(); run.read_file("app.py"); run.download("workspace.zip")
run.log(stage=1); run.log(stage=1, stream="stderr"); run.server_log()
run.requests()           # each model request through the proxy
run.gpu()                # GPU telemetry during the run
run.retry(); run.regrade(); run.cancel()

Local models take the GPU one at a time; cloud runs go in parallel (cloud_concurrency=). The process stays alive until a started plan finishes or is cancelled.

Following a plan live

watch() yields the same events the dashboard shows, until the plan (or run, or job) stops:

plan = llmdyno.bench(qwen, "suite_quick", wait=False)
for ev in llmdyno.watch(plan):
    if ev.kind == "run" and ev.finished:
        print(ev.data["benchmark_id"], ev.status, ev.data["correctness"])
    elif ev.kind == "gpu":
        print("VRAM", ev.data.get("vram_used_mib"))

Event kinds: run (a run changed state), plan, gpu (telemetry samples while a local model runs), job and log (background jobs such as downloads), and with watch(plan, transcript=True) also transcript (each agent event) and proxy (each model request with tokens, cost and latency). Or pass a callback to the blocking calls: llmdyno.bench(..., on_event=print).

Events arrive live for work started in the same Python process. Work running elsewhere, such as a plan started from the dashboard of a separate llmdyno serve, is followed by polling: run and plan changes only, every poll_s seconds.

Async code and notebooks

llmdyno.aio has every function as a coroutine (each runs in a worker thread), plus an async watch:

from llmdyno import aio

plan = await aio.bench(qwen, "suite_quick", wait=False)
async for ev in aio.watch(plan):
    ...
results = await aio.call(plan.results)      # any blocking method, in a thread

Results

print(results)                  # leaderboard, plus SWE-bench / polyglot rates when there are such runs
results.leaderboard()           # one dict per profile × harness, best first (scores 0-100)
results.runs()                  # one dict per run
results.public()                # SWE-bench resolved % and polyglot solved % per configuration
results.best()

results.to_pandas()             # DataFrame (pip install pandas); to_pandas("runs") for every run
results.to_csv("leaderboard.csv")
results.to_json("results.json")
results.open()                  # the full dashboard: charts, transcripts, workspaces

llmdyno.results(plans=[...], correctness=100, quality=0, instruction_following=0, efficiency=0)   # re-weight
plan.swebench_predictions("preds.jsonl")   # SWE-bench patches in the official predictions format

Every option and view of the Results page:

r = llmdyno.results(plan, benchmarks=["t5_bugfix_semver"], pass_threshold=1.0,
                    quality_weights={"static": 20, "probes": 30, "review": 50}, efficiency_ref="all")
r.cells()          # profile × harness × benchmark, with spread over iterations (the heatmap)
r.rubric()         # average judge score per rubric dimension
r.agreement()      # how much the judges agree, and self-preference
r.reviews()
r.reference()      # Artificial Analysis coding index next to each score
r.export("runs.csv", level="runs")          # the page's export: level runs / cells / configs, fmt csv / json
llmdyno.save_weights(correctness=60, quality=20)   # the defaults results() and the page start from
llmdyno.load_sample(); llmdyno.unload_sample()     # the bundled round-1 results, to explore without running anything

In Jupyter, a Results object renders as a table.

Code review by LLM judges

judge = llmdyno.cloud_model("anthropic", "claude-sonnet-5-5", name="Judge: Sonnet")
results = llmdyno.review(plan, judges=[judge], budget_usd=0.50)

Blind, shuffled review with LLMDyno's rubric; the scores feed the code-quality part of the final score.

llmdyno.judge_defaults()                         # the rubric, guidance and default limits
llmdyno.review(judges=[judge], runs=[run_id], rubric=[...], guidance="Be strict about tests",
               include_static=True, skip_reviewed=False)
llmdyno.reviews(plan=plan); llmdyno.reviews(run=run_id)
llmdyno.delete_reviews(judge, plan=plan)

Artificial Analysis reference

llmdyno.set_aa_key("...")        # free key from artificialanalysis.ai
llmdyno.refresh_aa()             # cached for a day; the free tier allows 100 requests a day
llmdyno.aa_reference()           # matches per profile and the model list
llmdyno.set_aa_match(qwen, "qwen3-8-27b")   # pick the match by hand (None: back to automatic)

Settings, system and jobs

s = llmdyno.settings()
s.cloud_concurrency, s.weights, s.data_dir
s.update(cloud_concurrency=4, electricity_usd_per_kwh=0.30)
llmdyno.set_hf_token("hf_..."); llmdyno.remove_hf_token()

llmdyno.specs(); llmdyno.gpu_usage()
llmdyno.list_jobs(active=True); llmdyno.job(job_id).wait(); llmdyno.clear_jobs()
llmdyno.active_work(); llmdyno.panic()           # the red button: stop everything
llmdyno.cleanup()                                # remove containers left by crashed runs
llmdyno.start_docker(); llmdyno.repair_docker(); llmdyno.use_llama_cpp("D:/llama.cpp")

Command line

Every call above is also a command, generated from the library so the two always match. Function names become commands with dashes, methods become llmdyno <object> <method> ID, and --json gives machine-readable output:

llmdyno list-models --status ready
llmdyno estimate unsloth/Qwen3.8-27B-GGUF UD-Q3_K_XL --set ctx=65536 --set kv_cache=q8_0
llmdyno bench "Qwen 27B 32k" --benchmarks suite_quick --harnesses pi opencode
llmdyno watch PLAN_ID
llmdyno results --plans PLAN_ID --view cells --json
llmdyno run transcript RUN_ID --view tool-calls
llmdyno plan retry PLAN_ID
llmdyno --help; llmdyno run --help; llmdyno bench --help

llmdyno alone starts the web app, as before. Errors print to stderr and exit with code 1. Two commands have no Python twin: llmdyno ci (fail a build on a score drop, see INTEGRATIONS.md) and llmdyno token (the access token for remote clients, see REMOTE.md).

Integrations and remote mode

Data frames, charts, notebooks, MLflow, Weights & Biases, Langfuse, OpenTelemetry, LiteLLM, Ollama, the Hugging Face Hub, your own LangGraph / OpenAI Agents SDK / smolagents / PydanticAI agent, Inspect AI and pytest: see INTEGRATIONS.md. To run LLMDyno on a GPU box and use it from your laptop, see REMOTE.md.

The dashboard

llmdyno.serve()                    # blocks; Ctrl+C stops it
llmdyno.serve(block=False)         # background thread, handy in notebooks

Data folder

By default the library uses the same folder as the app: LLMDYNO_DATA if set, otherwise ./data in a git checkout, %LOCALAPPDATA%\LLMDyno on Windows, ~/Library/Application Support/LLMDyno on macOS and ~/.local/share/llmdyno on Linux. To use another one:

llmdyno.init("~/llm-benchmarks")
llmdyno.data_dir()

Running plans from a script and from the app at the same time on one machine is not supported: both would try to load models onto the same GPU.

Errors

The library and the web API share one error family (llmdyno.errors); each carries the HTTP status the API answers with, so an error means the same thing from Python and over REST.

Exception When HTTP
llmdyno.PreflightError a plan cannot start; .problems lists why, .report has the full preflight 409
llmdyno.NotFoundError no such profile, benchmark, plan, run, file or quant 404
llmdyno.ConflictError a name or file is ambiguous, already in use, or in the wrong state 409
llmdyno.InvalidError a request that cannot be valid as given (also a ValueError) 400
llmdyno.ForbiddenError not allowed, such as editing a built-in benchmark 403
llmdyno.RiskNotAcknowledgedError danger warnings need acknowledge_risks=True 428
llmdyno.RateLimitError, UpstreamError an external service limited or failed the request 429, 502
llmdyno.InsufficientStorageError not enough disk space, for example for a download 507
llmdyno.LLMDynoError anything else LLMDyno refuses or fails to do (all of the above inherit from it) 400
TypeError an unknown profile setting

API at a glance

Call Returns
init(data_dir=None), data_dir() the data folder
doctor(), setup(...) Doctor (list of checks with .ok, .missing)
cloud_model(vendor, model, ...), local_model(repo, quant=..., ...), profile(name), list_profiles(), import_profile(path) Profile
profile_defaults(kind, model=) dict
search_models(q), repo_info(repo), estimate(repo, quant, ...), recommended_models() lists and dicts
download_model(repo, quant), model(name), list_models(), import_models(folder) Model
provider(name), list_providers(), add_provider(vendor, ...) Provider
vendors(), lookup_price(vendor, model) lists and dicts
harness(id), list_harnesses(), add_harness(manifest) Harness
list_benchmarks(), list_suites(), import_catalog() lists and dicts
benchmark(id), create_benchmark(spec), import_benchmark(zip) Benchmark
suite(id), create_suite(name, benchmarks) Suite
import_swebench(), import_polyglot(), install_swebench_harness(), build_polyglot_image(), remove_imported(kind) job results
preflight(models, benchmarks, harnesses) dict
bench(models, benchmarks, harnesses, ...) Results (or Plan with wait=False)
list_plans(), plan(id) Plan
run(id), plan.get_runs() Run; run.transcript() gives a Transcript
results(plans, weights, benchmarks=, ...), review(plans, judges, ...) Results
judge_defaults(), reviews(plan=, run=), delete_reviews(judge) dicts, lists, counts
save_weights(...), load_sample(), unload_sample(), sample_status() dicts
aa_reference(), refresh_aa(), set_aa_key(), remove_aa_key(), set_aa_match(profile, slug) dicts
settings() Settings (.update(...)); set_hf_token(), remove_hf_token()
specs(), gpu_usage(), active_work(), panic(), cleanup() dicts, counts
list_jobs(), job(id), clear_jobs() Job (.status, .progress, .log, .wait(), .cancel())
start_docker(), repair_docker(), install_docker(), use_llama_cpp(path) messages
watch(plan or run or job, transcript=False) iterator of Event
python_harness("module:function", requirements=[...]) Harness
node_harness("agent.ts:run", packages=[...]) Harness
local_servers(), ollama_model(), openai_compatible(), litellm_model() lists, Profile
compare_results(current, baseline) regression report
inspect_task(), import_inspect_samples() Inspect Task, Suite
load_results_from_hub(repo) Results
connect(url, token), disconnect(), connection() the backend in use
serve() dashboard URL
llmdyno.aio.<any function>, aio.watch(), aio.call(fn, ...) the same, awaitable

Every object (Profile, Plan, Job, …) gives its raw data with obj["field"], obj.get("field") and obj.to_dict(), and obj.refresh() fetches it again.

API stability

LLMDyno follows Semantic Versioning. Everything in llmdyno.__all__ (and the methods of the objects it returns) is the public API; modules and names that start with an underscore, and everything under llmdyno.services, llmdyno.api and the engine packages, are internal.

  • Until 1.0, a minor release (0.5 → 0.6) may change the public API; the CHANGELOG says what changed and how to adapt.
  • From 1.0, a public name is only removed or changed incompatibly in a major release. It is deprecated first: it keeps working for at least one minor release and warns (DeprecationWarning) with what to use instead.
  • Results stay comparable: a result records the LLMDyno version that computed its scores, and scoring changes are listed in the CHANGELOG.
  • The web API follows the library: each route has a library call (tests/test_parity.py), so a route changes when its call does.

Library and web app parity

Everything the dashboard can do, the library can do. tests/test_parity.py checks every web API route against the library (each library call names the routes it covers with @covers in llmdyno/sdk), and tests/test_cli.py checks that every library call has its command, so a new feature reaches the dashboard, Python and the command line together. python -c "from llmdyno.sdk import parity; print(parity.report())" prints the map.