Skip to content

Qrels

A judge's create_qrels() returns Qrels — (topic_id, doc_id, grade) rows. Define extractor functions in a QrelsSpec, build with build_qrels, verify with Qrels.verify(...), and serialize to TREC format with write_qrel_file. For generated text without a corpus id, derive a stable id with doc_id_md5.

Qrels and rows

autojudge_base.qrels.Qrels dataclass 

Qrels(rows: Sequence[QrelRow])

Collection of relevance judgments.

Qrels are intentionally policy-free
  • doc_id is opaque
  • no assumptions about corpus vs generated content
  • no assumptions about how grades are produced

rows instance-attribute 

rows: Sequence[QrelRow]

verify 

verify(expected_topic_ids: Optional[Sequence[str]], warn: Optional[bool] = False)
Source code in src/autojudge_base/qrels/qrels.py 
43
44
def verify(self, expected_topic_ids: Optional[Sequence[str]], warn:Optional[bool]=False):
    QrelsVerification(self, expected_topic_ids=expected_topic_ids, warn=warn).all()

autojudge_base.qrels.QrelRow dataclass 

QrelRow(topic_id: str, doc_id: str, grade: int)

topic_id instance-attribute 

topic_id: str

doc_id instance-attribute 

doc_id: str

grade instance-attribute 

grade: int

Spec and builder

autojudge_base.qrels.QrelsSpec dataclass 

QrelsSpec(topic_id: Callable[[R], str], doc_id: Callable[[R], str], grade: Callable[[R], int], on_duplicate: OnDuplicate = 'error')

Bases: Generic[R]

topic_id instance-attribute 

topic_id: Callable[[R], str]

doc_id instance-attribute 

doc_id: Callable[[R], str]

grade instance-attribute 

grade: Callable[[R], int]

on_duplicate class-attribute instance-attribute 

on_duplicate: OnDuplicate = 'error'

autojudge_base.qrels.build_qrels 

build_qrels(*, records: Iterable[R], spec: QrelsSpec[R]) -> list[QrelRow]
Source code in src/autojudge_base/qrels/qrels.py 
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
def build_qrels(*, records: Iterable[R], spec: QrelsSpec[R]) -> list[QrelRow]:
    seen: dict[tuple[str, str], int] = {}
    for r in records:
        tid = spec.topic_id(r)
        did = spec.doc_id(r)
        g = int(spec.grade(r))

        key = (tid, did)
        if key in seen:
            if spec.on_duplicate == "error":
                raise ValueError(f"Duplicate qrel for {key}: old={seen[key]} new={g}")
            elif spec.on_duplicate == "keep_max":
                seen[key] = max(seen[key], g)
            elif spec.on_duplicate == "keep_last":
                seen[key] = g
            else:
                raise ValueError(f"Unknown on_duplicate: {spec.on_duplicate}")
        else:
            seen[key] = g

    return Qrels(rows=[QrelRow(topic_id=tid, doc_id=did, grade=g) for (tid, did), g in seen.items()])

Verification and I/O

autojudge_base.qrels.QrelsVerification 

QrelsVerification(qrels: Qrels, expected_topic_ids: Sequence[str], warn: Optional[bool] = False)

Fluent verifier for qrels.

Chain verification methods to run multiple checks:

QrelsVerification(qrels, expected_topic_ids).complete_topics().no_duplicates()

Or run all checks:

QrelsVerification(qrels, expected_topic_ids).all()

Each method raises QrelsVerificationError on failure (fail-fast).

Initialize verifier.

Parameters:

Name Type Description Default
qrels Qrels

The qrels to verify

 required
expected_topic_ids Sequence[str]

The expected topic IDs to verify against

 required
warn Optional[bool]

If True, print warnings instead of raising exceptions

 False
Source code in src/autojudge_base/qrels/verification.py 
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
def __init__(
    self,
    qrels: "Qrels",
    expected_topic_ids: Sequence[str],
    warn: Optional[bool]=False
):
    """
    Initialize verifier.

    Args:
        qrels: The qrels to verify
        expected_topic_ids: The expected topic IDs to verify against
        warn: If True, print warnings instead of raising exceptions
    """
    self.qrels = qrels
    self.expected_topic_ids = expected_topic_ids
    self.warn = warn

qrels instance-attribute 

qrels = qrels

expected_topic_ids instance-attribute 

expected_topic_ids = expected_topic_ids

warn instance-attribute 

warn = warn

complete_topics 

complete_topics() -> QrelsVerification

Verify every expected topic has at least one qrel row.

Raises:

Type Description
QrelsVerificationError

If any topic is missing qrels
Source code in src/autojudge_base/qrels/verification.py 
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
def complete_topics(self) -> "QrelsVerification":
    """
    Verify every expected topic has at least one qrel row.

    Raises:
        QrelsVerificationError: If any topic is missing qrels
    """
    expected = set(self.expected_topic_ids)
    seen = set()

    for r in self.qrels.rows:
        if r.topic_id in expected:
            seen.add(r.topic_id)

    missing = expected - seen
    if missing:
        missing_list = sorted(missing)
        raise QrelsVerificationError(
            f"Missing qrels for {len(missing_list)} topic(s): {format_preview(missing_list)}"
        )

    return self

no_extra_topics 

no_extra_topics() -> QrelsVerification

Verify no qrels exist for non-expected topics.

Raises:

Type Description
QrelsVerificationError

If qrels exist for unknown topics
Source code in src/autojudge_base/qrels/verification.py 
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
def no_extra_topics(self) -> "QrelsVerification":
    """
    Verify no qrels exist for non-expected topics.

    Raises:
        QrelsVerificationError: If qrels exist for unknown topics
    """
    expected = set(self.expected_topic_ids)
    extras = {r.topic_id for r in self.qrels.rows} - expected

    if extras:
        extra_list = sorted(extras)
        raise QrelsVerificationError(
            f"Qrels for {len(extra_list)} unexpected topic(s): {format_preview(extra_list)}"
        )

    return self

no_duplicates 

no_duplicates() -> QrelsVerification

Verify no duplicate (topic_id, doc_id) pairs exist.

Raises:

Type Description
QrelsVerificationError

If duplicates are found
Source code in src/autojudge_base/qrels/verification.py 
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
def no_duplicates(self) -> "QrelsVerification":
    """
    Verify no duplicate (topic_id, doc_id) pairs exist.

    Raises:
        QrelsVerificationError: If duplicates are found
    """
    checked: Dict[Tuple[str, str], "QrelRow"] = {}

    for r in self.qrels.rows:
        key = (r.topic_id, r.doc_id)
        if key in checked:
            raise QrelsVerificationError(
                f"Duplicate qrel for topic={r.topic_id} doc_id={r.doc_id}: "
                f"first={checked[key]}, duplicate={r}"
            )
        checked[key] = r

    return self

all 

all() -> QrelsVerification

Run all verification checks.

Checks run in order (fail-fast): 1. no_duplicates - no duplicate (topic_id, doc_id) pairs 2. complete_topics - every expected topic has qrels 3. no_extra_topics - no qrels for unknown topics

Returns:

Type Description
QrelsVerification

self for chaining
Source code in src/autojudge_base/qrels/verification.py 
118
119
120
121
122
123
124
125
126
127
128
129
130
def all(self) -> "QrelsVerification":
    """
    Run all verification checks.

    Checks run in order (fail-fast):
    1. no_duplicates - no duplicate (topic_id, doc_id) pairs
    2. complete_topics - every expected topic has qrels
    3. no_extra_topics - no qrels for unknown topics

    Returns:
        self for chaining
    """
    return self.no_duplicates().complete_topics().no_extra_topics()

autojudge_base.qrels.QrelsVerificationError

Bases: Exception

Raised when qrels verification fails.

autojudge_base.qrels.write_qrel_file 

write_qrel_file(*, qrel_out_file: Union[str, Path], qrels: Qrels, _ignored: str = '0') -> None

Write qrels in standard TREC format:

topic_id  iteration  doc_id  grade

Notes: - The iteration field is always '0' (historical artifact, ignored by TREC tools). - Ordering is deterministic (sorted by topic_id, then doc_id).

Source code in src/autojudge_base/qrels/qrels.py 
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
def write_qrel_file(
    *,
    qrel_out_file: Union[str, Path],
    qrels: Qrels,
    _ignored: str = "0",
) -> None:
    """
    Write qrels in standard TREC format:

        topic_id  iteration  doc_id  grade

    Notes:
    - The iteration field is always '0' (historical artifact, ignored by TREC tools).
    - Ordering is deterministic (sorted by topic_id, then doc_id).
    """
    path = Path(qrel_out_file)
    path.parent.mkdir(parents=True, exist_ok=True)

    # Sort for reproducibility
    rows = sorted(
        qrels.rows,
        key=lambda r: (r.topic_id, r.doc_id),
    )

    with path.open("w", encoding="utf-8") as f:
        for r in rows:
            f.write(f"{r.topic_id} {_ignored} {r.doc_id} {r.grade}\n")

autojudge_base.qrels.doc_id_md5 

doc_id_md5(text: str) -> str
Source code in src/autojudge_base/qrels/qrels.py 
12
13
def doc_id_md5(text: str) -> str:
    return hashlib.md5(text.encode("utf-8")).hexdigest()