Results¶
Scores, every view of the Results page, exports and charts.
results ¶
Scored results: the leaderboard, per-run rows, public-benchmark rates and exports.
Results ¶
Results(data: dict[str, Any], plan_ids: Sequence[str] = (), options: dict[str, Any] | None = None)
Scored results. print(results) shows the leaderboard; leaderboard() and runs() return rows; to_pandas(), to_csv() and to_json() export them; open() shows the dashboard.
plot
property
¶
plot
Charts like the dashboard's: results.plot.leaderboard(), .heatmap(), .pareto(), .radar(), .tokens(); needs "llmdyno[plot]".
cells ¶
cells() -> list[Any]
One row per profile × harness × benchmark: pass rate, scores with standard deviation and 95% interval over iterations, time, tokens and cost (the heatmap).
reviews ¶
reviews() -> list[Any]
Every LLM-judge review behind these results (scores per rubric dimension, summary, issues).
rubric ¶
rubric() -> list[Any]
Average judge score per rubric dimension for each profile × harness (the rubric chart).
agreement ¶
agreement() -> dict[str, Any]
How much the judges agree with each other (correlation per judge pair) and how much each favours its own model family: {"judges", "pairs", …}.
reference ¶
reference() -> list[Any]
Each profile's Artificial Analysis coding index next to its score here (needs set_aa_key()).
export ¶
export(path: Union[str, Path, None] = None, level: str = 'runs', fmt: str = 'csv') -> str
The Results page's export: level "runs", "cells" or "configs", fmt "csv" or "json" (with the weights and LLMDyno version recorded). Written to path if given; returns the text.
leaderboard ¶
leaderboard() -> list[Any]
One row per profile × harness, best first. Scores are 0-100.
public ¶
public() -> list[Any]
SWE-bench resolved rate and Aider polyglot solved rate per profile × harness (only when such runs exist).
rows ¶
rows(level: str = 'configs') -> list[Any]
The rows of one table: configs (the leaderboard), runs, cells, reviews, rubric or public.
to_pandas ¶
to_pandas(level: str = 'configs') -> Any
A pandas DataFrame of one table (see rows()): the leaderboard by default.
to_polars ¶
to_polars(level: str = 'configs') -> Any
A polars DataFrame of one table (see rows()); needs "llmdyno[data]".
to_parquet ¶
to_parquet(path: Union[str, Path], level: str = 'runs') -> Path
Write one table to a Parquet file; needs "llmdyno[data]".
log_to_mlflow ¶
log_to_mlflow(experiment: str = 'llmdyno', tracking_uri: str | None = None, transcripts: bool = True) -> list[Any]
One MLflow run per configuration: settings as params, scores as metrics, results and transcripts as artifacts. Returns the MLflow run ids; needs "llmdyno[mlflow]".
log_to_wandb ¶
log_to_wandb(project: str = 'llmdyno', entity: str | None = None, name: str | None = None, mode: str | None = None, dir: str | None = None, transcripts: bool = True) -> str
The leaderboard, configurations and runs as W&B tables, transcripts as an artifact (mode="offline" works without an account). Returns the W&B run id; needs "llmdyno[wandb]".
log_to_langfuse ¶
log_to_langfuse(client: Any = None, public_key: str | None = None, secret_key: str | None = None, host: str | None = None) -> list[Any]
Each run as a Langfuse trace with one generation per model request and the grader scores. Returns the trace ids; needs "llmdyno[langfuse]".
to_otel ¶
to_otel(tracer_provider: Any = None, endpoint: str | None = None, headers: dict[str, Any] | None = None, service_name: str = 'llmdyno') -> int
Each run as an OpenTelemetry trace (GenAI semantic conventions) for any OTLP backend. Returns the number of spans; needs "llmdyno[otel]".
push_to_hub ¶
push_to_hub(repo_id: str, private: bool = True, token: str | None = None) -> str
Publish these results as a Hugging Face dataset (Parquet tables, a dataset card, the full results JSON); private unless you say otherwise. Returns its URL; needs "llmdyno[hub]".
to_inspect_log ¶
to_inspect_log(path: Union[str, Path], task: str = 'llmdyno') -> Path
Write these results as an Inspect AI eval log (one sample per run) to open with inspect view;
needs "llmdyno[inspect]".
open ¶
open() -> str
Open these results in the dashboard (starts the web app in the background if it is not running).
results ¶
results(plans: Any = None, *, include_sample: bool = False, correctness: float | None = None, instruction_following: float | None = None, quality: float | None = None, efficiency: float | None = None, quality_weights: dict[str, Any] | None = None, benchmarks: Any = None, pass_threshold: float = 0.8, efficiency_ref: str = 'local') -> 'Results'
Scored results for some plans (default: every plan except the bundled sample), with every option of the Results page: the four weights reshape the final score (defaults 50/15/25/10); quality_weights splits code quality between "static", "probes" and "review"; benchmarks limits it to some benchmarks; pass_threshold is the correctness that counts as a pass (pass@k); efficiency_ref compares speed and cost among "local" or "all" runs.
save_weights ¶
save_weights(*, correctness: float | None = None, instruction_following: float | None = None, quality: float | None = None, efficiency: float | None = None, quality_weights: dict[str, Any] | None = None) -> dict[str, Any]
Save the default score weights (what results() and the Results page start from). The four main weights are relative (defaults 50/15/25/10); quality_weights splits code quality between "static", "probes" and "review". Returns the saved {"weights", "quality_weights"}.
sample_status ¶
sample_status() -> dict[str, Any]
{"available", "loaded", "plan_id"} for the bundled sample results (15 configurations measured on an RTX 5080).
load_sample ¶
load_sample() -> str
Load the bundled sample results into this data folder to explore the dashboard and the API without running anything; returns its plan id. results() leaves it out unless you pass include_sample=True or its plan id.
load_results_from_hub ¶
load_results_from_hub(repo_id: str, revision: str | None = None, token: str | None = None) -> 'Results'
Results someone published with Results.push_to_hub(); needs "llmdyno[hub]".