Skip to content

Integrations

LLMDyno works with the tools you already use. Each integration is an optional extra, so the core install stays small:

pip install "llmdyno[plot]"          # or data, notebook, mlflow, wandb, langfuse, otel, litellm, hub, inspect
pip install "llmdyno[all]"           # everything

Nothing is imported until you use it. If an extra is missing, the error names the one to install.

Extra Gives you
data to_pandas(), to_polars(), to_parquet() for every table
notebook a live widget for a running plan, progress bars
plot the dashboard's charts in matplotlib or plotly
mlflow, wandb, langfuse, otel experiment tracking and tracing
litellm, hub models behind LiteLLM, publishing results to Hugging Face
inspect Inspect AI tasks, datasets and logs

Benchmarking your own Python or TypeScript agent (python_harness, node_harness) and the pytest plugin need no extra.

Data frames

Every table of a Results comes out as a data frame or a Parquet file:

r = llmdyno.results(plan)
r.to_pandas()                 # the leaderboard; or "runs", "cells", "reviews", "rubric", "public"
r.to_polars("cells")
r.to_parquet("runs.parquet", level="runs")
r.rows("cells")               # the same rows as plain dicts

t = llmdyno.run(run_id).transcript()
t.to_pandas()                 # one row per model call: tokens, time to first token, cost, tool calls

Notebooks

Results, Plan and Run render as tables with score bars in Jupyter. A running plan has a live panel:

plan = llmdyno.bench(qwen, "suite_quick", wait=False)
plan.live()                   # progress, finished runs with scores, GPU readings, updating as it runs

bench() and Plan.wait() show a progress bar when tqdm is installed and you are in a terminal or a notebook. Pass progress=False to turn it off. llmdyno.aio has every call as a coroutine, for notebooks that run their own event loop.

Charts

The dashboard's charts, in its colours, with matplotlib (default) or plotly:

r.plot.leaderboard()                       # stacked points per configuration, best on top
r.plot.heatmap("correctness")              # configuration × benchmark
r.plot.pareto(x="cost_usd_total")          # or wall_s_total, peak_vram_mib, decode_tps
r.plot.radar()                             # the judges' rubric
r.plot.tokens()
r.plot.leaderboard(backend="plotly").show()

Each method returns the figure, so you can save it or restyle it.

Experiment tracking and tracing

r.log_to_mlflow(experiment="coding-agents", tracking_uri="http://localhost:5000")
r.log_to_wandb(project="coding-agents")                  # mode="offline" needs no account
r.log_to_langfuse()                                      # LANGFUSE_PUBLIC_KEY / _SECRET_KEY / _HOST
r.to_otel(endpoint="http://localhost:6006/v1/traces")    # any OTLP backend: Phoenix, Datadog, Honeycomb, Tempo, …
  • MLflow: one run per configuration. Params are the profile's settings, model and harness. Metrics are the scores, time, cost, speed and VRAM. Artifacts are the result JSON and each run's transcript.
  • Weights & Biases: the leaderboard, configurations and runs as tables, summary metrics for the best configuration, and the transcripts as an artifact.
  • Langfuse: each run is a trace, and each model request is a generation with its messages, reply, usage and cost. Grader scores become Langfuse scores. Langfuse stamps observations with the time you log them, and the real times are in metadata. For an exact timeline, send the same runs with to_otel() to Langfuse's OTLP endpoint.
  • OpenTelemetry: each run is a trace with one span per model request. Spans use the GenAI semantic conventions (gen_ai.usage.input_tokens, gen_ai.request.model, …) and keep the real start and end times.

To send results as soon as a plan finishes, use bench(..., auto_log="mlflow"). It also takes a list, or a dict of options such as {"wandb": {"mode": "offline"}}.

Model gateways and servers

llmdyno.local_servers()                          # Ollama, LM Studio, vLLM, SGLang, llama.cpp, LiteLLM on this machine
qwen = llmdyno.ollama_model("qwen3-coder:30b", ctx=65536)
lm = llmdyno.openai_compatible("http://localhost:1234/v1", "qwen3-coder-30b")
claude = llmdyno.litellm_model("bedrock/anthropic.claude-sonnet-4-5", reasoning_effort="low")

litellm_model() uses a LiteLLM proxy you start yourself, for example litellm --model bedrock/... or litellm --config config.yaml (needs pip install "litellm[proxy]"). That puts Azure OpenAI, Bedrock, Vertex AI and over a hundred other providers behind one endpoint. Prices come from LiteLLM's own table when the litellm package is installed, and budgets cap spend as for any cloud provider. The proxy's master key, if it has one, goes in api_key= or LITELLM_API_KEY.

Hugging Face Hub

r.push_to_hub("you/llmdyno-results")                       # private by default
r2 = llmdyno.load_results_from_hub("you/llmdyno-results")  # a Results object again

The dataset has Parquet tables (configs, runs, cells), a dataset card with the leaderboard, and the full results JSON. With huggingface_hub installed, LLMDyno also uses the token from hf auth login for downloads when you have not saved one in LLMDyno.

Your own agent

python_harness() turns a Python function into a harness. Your agent then runs in the same offline sandbox as the built-in harnesses, reaching the model only through LLMDyno's proxy. Its transcripts, token counts, budgets and grading work the same way.

h = llmdyno.python_harness("my_agent:run", requirements=["langgraph==1.2.24", "langchain-openai==1.8.5"])
llmdyno.bench(qwen, "suite_quick", harnesses=[h.id, "pi"])

The function is called once per stage, in the task's workspace. It receives any of these it declares: prompt, model, base_url (the proxy, OpenAI-compatible), api_key, workspace, stage, ctx, max_tokens, reasoning. It may be async. Your code is built into the image together with the pinned requirements.

Ready-made agents are in examples/agents. Each one was checked end to end against a scripted model:

Framework Template
LangGraph langgraph_agent.py
OpenAI Agents SDK openai_agents_agent.py
smolagents smolagents_agent.py
PydanticAI pydantic_ai_agent.py

A TypeScript agent works the same way with node_harness("agent.ts:run"). Templates for the Vercel AI SDK, Mastra, LangGraph.js and the OpenAI Agents SDK are in examples/agents/ts. See JavaScript and TypeScript.

Inspect AI

from inspect_ai import eval
task = llmdyno.inspect_task(["t0_smoke_cli", "t5_bugfix_semver"], profile="Qwen 27B 32k")
eval(task, model="none/none")                    # LLMDyno runs the agent; Inspect's own model is not called

suite = llmdyno.import_inspect_samples(dataset, prefix="gsm8k")   # Inspect samples as LLMDyno benchmarks
r.to_inspect_log("logs/llmdyno.eval")            # open with `inspect view`

Imported samples ask the agent to write its answer to answer.txt, and the grader checks that file contains the sample's target.

CI and pytest

Fail a build when a model, prompt or harness change makes things worse:

llmdyno results --plans PLAN --json > .llmdyno/baseline.json      # once, from a good run
llmdyno ci --baseline .llmdyno/baseline.json --plans NEW_PLAN --max-drop 2

llmdyno ci exits with 1 in two cases: a configuration's score (final by default, or --metric correctness, …) dropped by more than --max-drop points, or a configuration is missing (unless you pass --allow-missing). From Python, llmdyno.compare_results(current, baseline) returns the same report. examples/ci/llmdyno-regression.yml is a GitHub Actions workflow to start from.

The pytest plugin is installed with LLMDyno:

@pytest.mark.llmdyno(models="Qwen 27B 32k", benchmarks="t0_smoke_cli", min_score=80)
def test_smoke_still_passes(llmdyno_results):
    assert llmdyno_results.best()["correctness"] == 100

The marker runs the plan once per session for each set of arguments. The test fails when the best score is under min_score or min_correctness. Use --llmdyno-skip on machines without Docker, and --llmdyno-data DIR to pick the data folder.