Skip to content

API reference

Everything here is available as llmdyno.<name> after import llmdyno, and as a llmdyno command (see the command line). The pages follow the areas of the dashboard.

Page What is in it
Setup, system and jobs Opening a data folder, checking and installing dependencies, the machine's specs and GPU, background jobs, cleanup and the panic stop, and the dashboard.
Profiles A model plus the exact settings it is benchmarked with.
Models GGUF models: Hugging Face search, fit estimates, downloads and imports.
Providers Cloud providers, local OpenAI-compatible servers and LiteLLM.
Harnesses and agents Built-in and custom harnesses, and your own Python agent as a harness.
Benchmarks and suites Built-in, your own and imported benchmarks (SWE-bench, Aider polyglot), and suites.
Plans Running profiles × harnesses × benchmarks × iterations.
Runs and transcripts One run's details, transcript, workspace, logs and telemetry.
Results Scores, every view of the Results page, exports and charts.
Judging Blind code review by LLM judges.
Live events and async Following work as it happens.
Remote mode Driving a LLMDyno server on another machine.
Settings and references Settings, tokens and Artificial Analysis reference numbers.
Interop and CI Inspect AI, regression checks for CI and the pytest plugin.
Errors One error family for the library and the web API.

Integrations with other libraries are methods of these objects (Results.to_pandas(), Results.plot, Results.log_to_mlflow(), …) and are described in Integrations.