Module: Raif::Evals::EvalSets::LlmJudgeExpectations

Included in:
Raif::Evals::EvalSet
Defined in:
lib/raif/evals/eval_sets/llm_judge_expectations.rb

Instance Method Summary collapse

Instance Method Details

#expect_llm_judge_passes(content, criteria:, examples: [], strict: false, llm_judge_model_key: nil, additional_context: nil, result_metadata: {}, judge_attributes: {}, label: nil) ⇒ ExpectationResult

Note:

The judge result includes metadata accessible via expectation_result.metadata:

  • :passes - Boolean result
  • :reasoning - Detailed explanation
  • :confidence - Confidence score (0.0-1.0)

Uses an LLM judge to evaluate whether content meets specific criteria with a binary pass/fail result.

This method leverages the Binary LLM judge to assess content against provided criteria, returning a pass or fail judgment with reasoning and confidence scores.

Examples:

Basic usage

expect_llm_judge_passes(
  task.parsed_response,
  criteria: "Response is polite and professional"
)

With examples and strict mode

expect_llm_judge_passes(
  content,
  criteria: "Contains a proper greeting",
  examples: [
    { content: "Hello, how can I help?", passes: true, reasoning: "Contains greeting" },
    { content: "What do you want?", passes: false, reasoning: "No greeting, rude tone" }
  ],
  strict: true
)

Parameters:

  • content (String) —

    The content to be evaluated by the LLM judge

  • criteria (String) —

    The evaluation criteria that the content must meet

  • examples (Array<Hash>) (defaults to: []) —

    Optional examples showing how to evaluate similar content. Each example should have keys: :content, :passes (boolean), :reasoning

  • strict (Boolean) (defaults to: false) —

    Whether to apply criteria strictly (true) or with reasonable judgment (false)

  • llm_judge_model_key (Symbol, nil) (defaults to: nil) —

    Optional specific LLM model to use for judging. If nil, uses the configured default judge model or falls back to default LLM

  • additional_context (String, nil) (defaults to: nil) —

    Optional additional context to be provided to the judge

  • result_metadata (Hash) (defaults to: {}) —

    Optional metadata recorded on the expectation result, merged with the judge's own metadata

  • judge_attributes (Hash) (defaults to: {}) —

    Optional attributes merged into the judge task on top of the eval set's #judge_task_attributes

  • label (String, nil) (defaults to: nil) —

    Optional explicit description for the expectation, for when the criteria is too long to read well as one

Returns:

  • (ExpectationResult) —

    Result object containing pass/fail status and judge metadata



51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
 
# File 'lib/raif/evals/eval_sets/llm_judge_expectations.rb', line 51

def expect_llm_judge_passes(content, criteria:, examples: [], strict: false, llm_judge_model_key: nil, additional_context: nil,
  result_metadata: {}, judge_attributes: {}, label: nil)
  judge_task = LlmJudges::Binary.run(
    content_to_judge: content,
    criteria: criteria,
    examples: examples,
    strict_mode: strict,
    llm_model_key: llm_judge_model_key,
    additional_context: additional_context,
    **resolved_judge_attributes(judge_attributes)
  )

  # To console_output rather than output: a dataset run discards output so it can print one
  # compact line per case, and a low-confidence judgment is a reason to distrust that line.
  if judge_task.low_confidence?
    console_output.puts Raif::Utils::Colors.yellow("    ⚠ Low confidence: #{judge_task.judgment_confidence}")
  end

  output.puts "    #{judge_task.judgment_reasoning}" if Raif.config.evals_verbose_output

  judge_metadata = {
    passes: judge_task.passes?,
    reasoning: judge_task.judgment_reasoning,
    confidence: judge_task.judgment_confidence,
  }.compact

  # Merge user metadata with judge metadata
  combined_metadata = result_metadata.merge(judge_metadata)

  expectation_result = expect label || "LLM judge: #{criteria}", result_metadata: combined_metadata do
    judge_task.passes?
  end

  if expectation_result && judge_task.errors.any?
    expectation_result.error_message = judge_task.errors.full_messages.join(", ")
  end

  expectation_result
end

#expect_llm_judge_prefers(content_to_judge, over:, criteria:, allow_ties: true, llm_judge_model_key: nil, additional_context: nil, result_metadata: {}, judge_attributes: {}, label: nil) ⇒ ExpectationResult

Note:

The expectation passes if the judge correctly identifies the expected winner. Due to randomization, content_to_judge may be assigned to either position A or B, and the judge's choice is validated against the expected winner.

Note:

The judge result includes metadata accessible via expectation_result.metadata:

  • :winner - Which content won ("A", "B", or "tie")
  • :reasoning - Detailed explanation of the choice
  • :confidence - Confidence score (0.0-1.0)

Uses an LLM judge to compare two pieces of content and determine which better meets specified criteria.

This method leverages the Comparative LLM judge to perform A/B testing between two pieces of content. Content placement is randomized to avoid position bias, and the judge determines which content better satisfies the comparison criteria.

Examples:

Basic A/B comparison

expect_llm_judge_prefers(
  new_response,
  over: baseline_response,
  criteria: "More comprehensive and accurate response"
)

Model comparison with no ties allowed

expect_llm_judge_prefers(
  claude_response,
  over: gpt_response,
  criteria: "Better follows the specific instructions given",
  allow_ties: false
)

With additional context

expect_llm_judge_prefers(
  response_a,
  over: response_b,
  criteria: "More helpful for a beginner audience",
  additional_context: "The user identified themselves as new to programming"
)

Parameters:

  • content_to_judge (String) —

    The primary content being evaluated (will be randomly assigned to position A or B)

  • over (String) —

    The comparison content to evaluate against (will be randomly assigned to position A or B)

  • criteria (String) —

    The comparison criteria to use for evaluation

  • allow_ties (Boolean) (defaults to: true) —

    Whether the judge can declare a tie if both contents are equal (default: true)

  • llm_judge_model_key (Symbol, nil) (defaults to: nil) —

    Optional specific LLM model to use for judging. If nil, uses the configured default judge model or falls back to default LLM

  • additional_context (String, nil) (defaults to: nil) —

    Optional additional context to help the judge

  • result_metadata (Hash) (defaults to: {}) —

    Optional metadata recorded on the expectation result, merged with the judge's own metadata

  • judge_attributes (Hash) (defaults to: {}) —

    Optional attributes merged into the judge task on top of the eval set's #judge_task_attributes

  • label (String, nil) (defaults to: nil) —

    Optional explicit description for the expectation, for when the criteria is too long to read well as one

Returns:

  • (ExpectationResult) —

    Result object containing pass/fail status and judge metadata



272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
 
# File 'lib/raif/evals/eval_sets/llm_judge_expectations.rb', line 272

def expect_llm_judge_prefers(content_to_judge, over:, criteria:, allow_ties: true, llm_judge_model_key: nil, additional_context: nil,
  result_metadata: {}, judge_attributes: {}, label: nil)
  judge_task = LlmJudges::Comparative.run(
    content_to_judge: content_to_judge,
    over_content: over,
    comparison_criteria: criteria,
    allow_ties: allow_ties,
    llm_model_key: llm_judge_model_key,
    additional_context: additional_context,
    **resolved_judge_attributes(judge_attributes)
  )

  output.puts "    Winner: #{judge_task.winner}"
  output.puts "    #{judge_task.judgment_reasoning}" if Raif.config.evals_verbose_output

  judge_metadata = {
    winner: judge_task.winner,
    reasoning: judge_task.judgment_reasoning,
    confidence: judge_task.judgment_confidence,
  }.compact

  # Merge user metadata with judge metadata
  combined_metadata = result_metadata.merge(judge_metadata)

  expectation_result = expect label || "LLM judge prefers A over B: #{criteria}", result_metadata: combined_metadata do
    judge_task.completed? && judge_task.correct_expected_winner?
  end

  if expectation_result && judge_task.errors.any?
    expectation_result.error_message = judge_task.errors.full_messages.join(", ")
  end

  expectation_result
end

#expect_llm_judge_score(content, scoring_rubric:, min_passing_score:, llm_judge_model_key: nil, additional_context: nil, result_metadata: {}, judge_attributes: {}, label: nil, score_name: nil) ⇒ ExpectationResult

Note:

The judge result includes metadata accessible via expectation_result.metadata:

  • :score - Numerical score given
  • :reasoning - Detailed explanation
  • :confidence - Confidence score (0.0-1.0)

Uses an LLM judge to evaluate content with a numerical score based on a detailed rubric.

This method leverages the Scored LLM judge to assess content against a scoring rubric, providing a numerical score with detailed reasoning and determining pass/fail based on the minimum passing score threshold.

As well as the pass/fail expectation, the judge's score is recorded as a first-class score named after the rubric, so it lands in the run's score summaries and can be compared across runs.

Examples:

Using a built-in rubric

expect_llm_judge_score(
  task.parsed_response,
  scoring_rubric: ScoringRubric.accuracy,
  min_passing_score: 8
)

Using a custom rubric

rubric = ScoringRubric.new(
  name: :technical_writing,
  description: "Evaluates technical writing quality",
  levels: [
    { score_range: (9..10), description: "Expert-level technical content" },
    { score_range: (7..8), description: "Strong technical content" },
    { score_range: (5..6), description: "Adequate technical content" },
    { score_range: (3..4), description: "Weak technical content" },
    { score_range: (0..2), description: "Poor technical content" }
  ]
)
expect_llm_judge_score(output, scoring_rubric: rubric, min_passing_score: 7)

Using a simple string rubric, which must name its own score

expect_llm_judge_score(
  output,
  scoring_rubric: "Rate clarity from 0-5 where 5 is crystal clear",
  min_passing_score: 4,
  score_name: "clarity"
)

Parameters:

  • content (String) —

    The content to be evaluated by the LLM judge

  • scoring_rubric (ScoringRubric, String) —

    The rubric to use for scoring. Can be a ScoringRubric object with structured levels or a plain string description

  • min_passing_score (Integer) —

    Minimum score required to pass

  • llm_judge_model_key (Symbol, nil) (defaults to: nil) —

    Optional specific LLM model to use for judging. If nil, uses the configured default judge model or falls back to default LLM

  • additional_context (String, nil) (defaults to: nil) —

    Optional additional context to be provided to the judge

  • result_metadata (Hash) (defaults to: {}) —

    Optional metadata recorded on the expectation result, merged with the judge's own metadata

  • judge_attributes (Hash) (defaults to: {}) —

    Optional attributes merged into the judge task on top of the eval set's #judge_task_attributes

  • label (String, nil) (defaults to: nil) —

    Optional explicit description for the expectation

  • score_name (String, Symbol, nil) (defaults to: nil) —

    Name for the recorded score, defaulting to the rubric's name. Needed when one eval judges two things against the same rubric and you want them tracked apart, and required when scoring_rubric is a String, which has no name to derive one from

Returns:

  • (ExpectationResult) —

    Result object containing pass/fail status and judge metadata

Raises:

  • (ArgumentError) —

    When scoring_rubric is a String and no score_name is given



155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
 
# File 'lib/raif/evals/eval_sets/llm_judge_expectations.rb', line 155

def expect_llm_judge_score(content, scoring_rubric:, min_passing_score:, llm_judge_model_key: nil, additional_context: nil,
  result_metadata: {}, judge_attributes: {}, label: nil, score_name: nil)
  scoring_rubric_obj = scoring_rubric

  # A score name is the metric the run summary aggregates by and evals:compare joins on. A
  # ScoringRubric carries one; a string rubric does not, and a placeholder would put an
  # unidentifiable metric in the summary. Raised before the judge call so it costs nothing.
  if score_name.blank? && !scoring_rubric_obj.respond_to?(:name)
    raise ArgumentError, "expect_llm_judge_score with a #{scoring_rubric_obj.class} rubric requires score_name: to name the " \
      "score it records. A Raif::Evals::ScoringRubric supplies that name itself; a rubric passed as a string has none."
  end

  rubric_name = scoring_rubric_obj.respond_to?(:name) ? scoring_rubric_obj.name : "custom"
  resolved_score_name = score_name.presence || rubric_name

  # Also before the judge runs: a name already taken by an earlier score in this eval
  # raises, and finding out on the way back would cost a judge request.
  current_eval_result.ensure_score_name_available!(resolved_score_name)

  judge_task = LlmJudges::Scored.run(
    content_to_judge: content,
    scoring_rubric: scoring_rubric_obj,
    llm_model_key: llm_judge_model_key,
    additional_context: additional_context,
    **resolved_judge_attributes(judge_attributes)
  )

  output.puts "    Score: #{judge_task.judgment_score}"
  output.puts "    #{judge_task.judgment_reasoning}" if Raif.config.evals_verbose_output

  judge_metadata = {
    score: judge_task.judgment_score,
    reasoning: judge_task.judgment_reasoning,
    confidence: judge_task.judgment_confidence,
  }.compact

  # Merge user metadata with judge metadata
  combined_metadata = result_metadata.merge(judge_metadata)

  # Recorded directly rather than through #score, so the gating expectation below keeps
  # its existing description - the key evals:compare matches results on across runs.
  if judge_task.judgment_score
    current_eval_result.add_score(
      ScoreResult.new(
        name: resolved_score_name,
        value: judge_task.judgment_score,
        scale: (scoring_rubric_obj.scale if scoring_rubric_obj.respond_to?(:scale)),
        min: min_passing_score
      )
    )
  end

  expectation_result = expect label || "LLM judge score (#{rubric_name}): >= #{min_passing_score}",
    result_metadata: combined_metadata do
    judge_task.completed? && judge_task.judgment_score && judge_task.judgment_score >= min_passing_score
  end

  if expectation_result && judge_task.errors.any?
    expectation_result.error_message = judge_task.errors.full_messages.join(", ")
  end

  expectation_result
end