Config & protocol

An AutoJudge implements the AutoJudge protocol (judge, create_nuggets, create_qrels) and receives its LLM endpoint through the injected LlmConfigBase. The framework builds the config from environment variables (OPENAI_BASE_URL/OPENAI_MODEL/OPENAI_API_KEY/CACHE_DIR) — see Configure your LLM endpoint.

AutoJudge protocol

autojudge_base.AutoJudge

Bases: LeaderboardJudgeProtocol, QrelsCreatorProtocol, NuggetCreatorProtocol, Protocol

Combined protocol for judges that implement all three phases.

This is a convenience protocol for the common case where a single class handles nugget creation, qrels creation, and leaderboard generation.

For modular configurations, use the individual protocols: - LeaderboardJudgeProtocol - QrelsCreatorProtocol - NuggetCreatorProtocol

LLM config

autojudge_base.llm_config.LlmConfigBase dataclass

LlmConfigBase(model: str = 'gpt-4o-mini', cache_dir: Optional[Path] = None, api_key: Optional[str] = None, base_url: Optional[str] = None, raw: dict = dict())

Minimal LLM configuration implementation.

For full features (batching, transport, retries), judges can use the raw config dict to instantiate MinimaLlmConfig from the minima-llm package:

from minima_llm import MinimaLlmConfig, OpenAIMinimaLlm
full_config = MinimaLlmConfig.from_dict(llm_config.raw)
backend = OpenAIMinimaLlm(full_config)

model class-attribute instance-attribute

model: str = 'gpt-4o-mini'

cache_dir class-attribute instance-attribute

cache_dir: Optional[Path] = None

api_key class-attribute instance-attribute

api_key: Optional[str] = None

base_url class-attribute instance-attribute

base_url: Optional[str] = None

raw class-attribute instance-attribute

raw: dict = field(default_factory=dict)

from_env classmethod

from_env() -> LlmConfigBase

Load from environment variables.

Source code in src/autojudge_base/llm_config.py

@classmethod
def from_env(cls) -> "LlmConfigBase":
    """Load from environment variables."""
    # Note: Path("") is Path(".") and truthy, so the empty-string fallback
    # must be resolved BEFORE constructing the Path — otherwise an unset
    # CACHE_DIR silently enables caching in the current directory.
    cache = os.getenv("CACHE_DIR") or os.getenv("LLM_CACHE_DIR")
    return cls(
        model=os.getenv("OPENAI_MODEL", os.getenv("LLM_MODEL", "gpt-4o-mini")),
        cache_dir=Path(cache) if cache else None,
        api_key=os.getenv("OPENAI_API_KEY"),
        # OPENAI_API_BASE is the litellm-convention name for the same thing;
        # accept both so judges work under either injection convention.
        base_url=os.getenv("OPENAI_BASE_URL") or os.getenv("OPENAI_API_BASE"),
        raw={},
    )

autojudge_base.llm_config.LlmConfigProtocol

Bases: Protocol

Protocol for LLM configuration classes.

model instance-attribute

model: str

cache_dir instance-attribute

cache_dir: Optional[Path]

from_env classmethod

from_env() -> LlmConfigProtocol

Load configuration from environment variables.

Source code in src/autojudge_base/llm_config.py

@classmethod
def from_env(cls) -> "LlmConfigProtocol":
    """Load configuration from environment variables."""
    ...