Skip to content

Results

Scores, every view of the Results page, exports and charts.

results

Scored results: the leaderboard, per-run rows, public-benchmark rates and exports.

Results

Results(data: dict[str, Any], plan_ids: Sequence[str] = (), options: dict[str, Any] | None = None)

Scored results. print(results) shows the leaderboard; leaderboard() and runs() return rows; to_pandas(), to_csv() and to_json() export them; open() shows the dashboard.

plot property

plot

Charts like the dashboard's: results.plot.leaderboard(), .heatmap(), .pareto(), .radar(), .tokens(); needs "llmdyno[plot]".

cells

cells() -> list[Any]

One row per profile × harness × benchmark: pass rate, scores with standard deviation and 95% interval over iterations, time, tokens and cost (the heatmap).

reviews

reviews() -> list[Any]

Every LLM-judge review behind these results (scores per rubric dimension, summary, issues).

rubric

rubric() -> list[Any]

Average judge score per rubric dimension for each profile × harness (the rubric chart).

agreement

agreement() -> dict[str, Any]

How much the judges agree with each other (correlation per judge pair) and how much each favours its own model family: {"judges", "pairs", …}.

reference

reference() -> list[Any]

Each profile's Artificial Analysis coding index next to its score here (needs set_aa_key()).

export

export(path: Union[str, Path, None] = None, level: str = 'runs', fmt: str = 'csv') -> str

The Results page's export: level "runs", "cells" or "configs", fmt "csv" or "json" (with the weights and LLMDyno version recorded). Written to path if given; returns the text.

leaderboard

leaderboard() -> list[Any]

One row per profile × harness, best first. Scores are 0-100.

runs

runs() -> list[Any]

One row per run (benchmark × iteration).

public

public() -> list[Any]

SWE-bench resolved rate and Aider polyglot solved rate per profile × harness (only when such runs exist).

rows

rows(level: str = 'configs') -> list[Any]

The rows of one table: configs (the leaderboard), runs, cells, reviews, rubric or public.

to_pandas

to_pandas(level: str = 'configs') -> Any

A pandas DataFrame of one table (see rows()): the leaderboard by default.

to_polars

to_polars(level: str = 'configs') -> Any

A polars DataFrame of one table (see rows()); needs "llmdyno[data]".

to_parquet

to_parquet(path: Union[str, Path], level: str = 'runs') -> Path

Write one table to a Parquet file; needs "llmdyno[data]".

log_to_mlflow

log_to_mlflow(experiment: str = 'llmdyno', tracking_uri: str | None = None, transcripts: bool = True) -> list[Any]

One MLflow run per configuration: settings as params, scores as metrics, results and transcripts as artifacts. Returns the MLflow run ids; needs "llmdyno[mlflow]".

log_to_wandb

log_to_wandb(project: str = 'llmdyno', entity: str | None = None, name: str | None = None, mode: str | None = None, dir: str | None = None, transcripts: bool = True) -> str

The leaderboard, configurations and runs as W&B tables, transcripts as an artifact (mode="offline" works without an account). Returns the W&B run id; needs "llmdyno[wandb]".

log_to_langfuse

log_to_langfuse(client: Any = None, public_key: str | None = None, secret_key: str | None = None, host: str | None = None) -> list[Any]

Each run as a Langfuse trace with one generation per model request and the grader scores. Returns the trace ids; needs "llmdyno[langfuse]".

to_otel

to_otel(tracer_provider: Any = None, endpoint: str | None = None, headers: dict[str, Any] | None = None, service_name: str = 'llmdyno') -> int

Each run as an OpenTelemetry trace (GenAI semantic conventions) for any OTLP backend. Returns the number of spans; needs "llmdyno[otel]".

push_to_hub

push_to_hub(repo_id: str, private: bool = True, token: str | None = None) -> str

Publish these results as a Hugging Face dataset (Parquet tables, a dataset card, the full results JSON); private unless you say otherwise. Returns its URL; needs "llmdyno[hub]".

to_inspect_log

to_inspect_log(path: Union[str, Path], task: str = 'llmdyno') -> Path

Write these results as an Inspect AI eval log (one sample per run) to open with inspect view; needs "llmdyno[inspect]".

open

open() -> str

Open these results in the dashboard (starts the web app in the background if it is not running).

results

results(plans: Any = None, *, include_sample: bool = False, correctness: float | None = None, instruction_following: float | None = None, quality: float | None = None, efficiency: float | None = None, quality_weights: dict[str, Any] | None = None, benchmarks: Any = None, pass_threshold: float = 0.8, efficiency_ref: str = 'local') -> 'Results'

Scored results for some plans (default: every plan except the bundled sample), with every option of the Results page: the four weights reshape the final score (defaults 50/15/25/10); quality_weights splits code quality between "static", "probes" and "review"; benchmarks limits it to some benchmarks; pass_threshold is the correctness that counts as a pass (pass@k); efficiency_ref compares speed and cost among "local" or "all" runs.

save_weights

save_weights(*, correctness: float | None = None, instruction_following: float | None = None, quality: float | None = None, efficiency: float | None = None, quality_weights: dict[str, Any] | None = None) -> dict[str, Any]

Save the default score weights (what results() and the Results page start from). The four main weights are relative (defaults 50/15/25/10); quality_weights splits code quality between "static", "probes" and "review". Returns the saved {"weights", "quality_weights"}.

sample_status

sample_status() -> dict[str, Any]

{"available", "loaded", "plan_id"} for the bundled sample results (15 configurations measured on an RTX 5080).

load_sample

load_sample() -> str

Load the bundled sample results into this data folder to explore the dashboard and the API without running anything; returns its plan id. results() leaves it out unless you pass include_sample=True or its plan id.

unload_sample

unload_sample() -> None

Remove the sample plan, its runs, reviews and profiles.

load_results_from_hub

load_results_from_hub(repo_id: str, revision: str | None = None, token: str | None = None) -> 'Results'

Results someone published with Results.push_to_hub(); needs "llmdyno[hub]".