Interop and CI¶
Inspect AI, regression checks for CI and the pytest plugin.
interop ¶
Working with other tools: Inspect AI tasks and datasets, and regression checks for CI.
inspect_task ¶
inspect_task(benchmarks: Any, profile: Any, harness: str = 'pi', name: str = 'llmdyno', **bench_options: Any)
An Inspect AI Task running LLMDyno benchmarks (or suites) with this profile and harness; evaluate it with inspect_ai.eval(task, model="none/none"). Needs "llmdyno[inspect]".
import_inspect_samples ¶
import_inspect_samples(samples: Any, prefix: str = 'inspect', suite: str | None = None, answer_file: str = 'answer.txt', timeout_s: int = 600)
LLMDyno benchmarks from an Inspect AI dataset: the agent writes its answer to answer_file and the grader checks it holds the sample's target. Returns a Suite of them. Needs "llmdyno[inspect]".
compare_results ¶
compare_results(current: Any, baseline: Any, *, metric: str = 'final', max_drop: float = 2.0, allow_missing: bool = False) -> dict[str, Any]
Compare results with a baseline, configuration by configuration (profile × harness), for CI. current and
baseline are Results, plan ids, or results JSON files (Results.to_json(), llmdyno results --json).
A configuration regresses when its metric (final, correctness, quality, … on 0-100) drops by more than max_drop
points; one missing from current fails the check unless allow_missing. Returns {"ok", "rows", "regressions",
"missing", "new"}.
pytest_plugin ¶
pytest plugin, registered automatically when llmdyno is installed: model and agent regression tests in your suite.
import pytest
@pytest.mark.llmdyno(models="Qwen 27B 32k", benchmarks="t0_smoke_cli", harnesses="pi", min_score=80)
def test_smoke_still_passes(llmdyno_results):
assert llmdyno_results.best()["correctness"] == 100
The marker runs the plan once per distinct set of arguments in a session (tests sharing them share the run). A test
fails when the best final score is under min_score or the best correctness under min_correctness, so a marked test
needs no body at all. Other marker keywords go to llmdyno.bench(). Fixtures: llmdyno (the library, on
--llmdyno-data if given) and llmdyno_results. Options: --llmdyno-data DIR, --llmdyno-skip (skip marked tests, e.g. on
machines without Docker or a GPU). Nothing here imports more than pytest until a marked test runs.
llmdyno ¶
llmdyno(pytestconfig)
The llmdyno library, on the data folder given with --llmdyno-data (else the usual one).
llmdyno_results ¶
llmdyno_results(request)
The Results of the plan described by the test's @pytest.mark.llmdyno(...).
pytest_runtest_call ¶
pytest_runtest_call(item)
The score limits are part of the test: a regression is a failure, not a setup error.