Skip to content

Interop and CI

Inspect AI, regression checks for CI and the pytest plugin.

interop

Working with other tools: Inspect AI tasks and datasets, and regression checks for CI.

inspect_task

inspect_task(benchmarks: Any, profile: Any, harness: str = 'pi', name: str = 'llmdyno', **bench_options: Any)

An Inspect AI Task running LLMDyno benchmarks (or suites) with this profile and harness; evaluate it with inspect_ai.eval(task, model="none/none"). Needs "llmdyno[inspect]".

import_inspect_samples

import_inspect_samples(samples: Any, prefix: str = 'inspect', suite: str | None = None, answer_file: str = 'answer.txt', timeout_s: int = 600)

LLMDyno benchmarks from an Inspect AI dataset: the agent writes its answer to answer_file and the grader checks it holds the sample's target. Returns a Suite of them. Needs "llmdyno[inspect]".

compare_results

compare_results(current: Any, baseline: Any, *, metric: str = 'final', max_drop: float = 2.0, allow_missing: bool = False) -> dict[str, Any]

Compare results with a baseline, configuration by configuration (profile × harness), for CI. current and baseline are Results, plan ids, or results JSON files (Results.to_json(), llmdyno results --json). A configuration regresses when its metric (final, correctness, quality, … on 0-100) drops by more than max_drop points; one missing from current fails the check unless allow_missing. Returns {"ok", "rows", "regressions", "missing", "new"}.

pytest_plugin

pytest plugin, registered automatically when llmdyno is installed: model and agent regression tests in your suite.

import pytest

@pytest.mark.llmdyno(models="Qwen 27B 32k", benchmarks="t0_smoke_cli", harnesses="pi", min_score=80)
def test_smoke_still_passes(llmdyno_results):
    assert llmdyno_results.best()["correctness"] == 100

The marker runs the plan once per distinct set of arguments in a session (tests sharing them share the run). A test fails when the best final score is under min_score or the best correctness under min_correctness, so a marked test needs no body at all. Other marker keywords go to llmdyno.bench(). Fixtures: llmdyno (the library, on --llmdyno-data if given) and llmdyno_results. Options: --llmdyno-data DIR, --llmdyno-skip (skip marked tests, e.g. on machines without Docker or a GPU). Nothing here imports more than pytest until a marked test runs.

llmdyno

llmdyno(pytestconfig)

The llmdyno library, on the data folder given with --llmdyno-data (else the usual one).

llmdyno_results

llmdyno_results(request)

The Results of the plan described by the test's @pytest.mark.llmdyno(...).

pytest_runtest_call

pytest_runtest_call(item)

The score limits are part of the test: a regression is a failure, not a setup error.