Module: Raif::Evals::EvalSets::Matchers

Included in:
Raif::Evals::EvalSet
Defined in:
lib/raif/evals/eval_sets/matchers.rb

Overview

Matchers for evals with a known-correct answer, typically an EvalCase's expected.

Each one wraps #expect. What they add over a hand-written block is normalization applied the same way in every eval set, and metadata: a hand-written include? records only that it failed, where these record what was produced and what was wanted.

Constant Summary collapse

MAX_METADATA_LENGTH =

Both sides of a comparison land in the results JSON for every case of every repeat, and an untruncated response is most of the file.

500

Instance Method Summary collapse

Instance Method Details

#expect_exact_match(actual, expected, ignore_case: true, strip: true, label: nil, result_metadata: {}) ⇒ ExpectationResult

Compares a value against ground truth, ignoring the differences a case or whitespace change makes. Strings are normalized and compared; anything else is compared with ==, so a boolean or numeric answer is not coerced through to_s first.

Examples:

expect_exact_match(task.parsed_response, eval_case.expected["answer"])

Parameters:

  • actual (Object) —

    the value produced by whatever is under test

  • expected (Object) —

    the ground-truth value

  • ignore_case (Boolean) (defaults to: true) —

    downcase both sides before comparing (strings only)

  • strip (Boolean) (defaults to: true) —

    strip surrounding whitespace from both sides (strings only)

  • label (String, nil) (defaults to: nil) —

    replaces the expectation's description

  • result_metadata (Hash) (defaults to: {}) —

    merged into the recorded metadata

Returns:



35
36
37
38
39
40
41
42
43
44
 
# File 'lib/raif/evals/eval_sets/matchers.rb', line 35

def expect_exact_match(actual, expected, ignore_case: true, strip: true, label: nil, result_metadata: {})
  normalized_actual = normalize_matcher_string(actual, ignore_case: ignore_case, strip: strip)
  normalized_expected = normalize_matcher_string(expected, ignore_case: ignore_case, strip: strip)

  metadata = result_metadata.merge(matcher_metadata(actual, expected))

  expect label || "exact match", result_metadata: metadata do
    normalized_actual == normalized_expected
  end
end

#expect_includes(actual, expected, ignore_case: true, label: nil, result_metadata: {}) ⇒ ExpectationResult

Asserts that text appears in a value. An Array of expected texts requires all of them, which is the common shape of "the summary has to mention each of these".

Examples:

expect_includes(task.parsed_response, eval_case.expected["keywords"])

Parameters:

  • actual (Object) —

    the value produced by whatever is under test, read as text

  • expected (String, Array<String>) —

    the text, or every text, that must appear

  • ignore_case (Boolean) (defaults to: true) —

    compare with both sides downcased

  • label (String, nil) (defaults to: nil) —

    replaces the expectation's description

  • result_metadata (Hash) (defaults to: {}) —

    merged into the recorded metadata

Returns:



59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
 
# File 'lib/raif/evals/eval_sets/matchers.rb', line 59

def expect_includes(actual, expected, ignore_case: true, label: nil, result_metadata: {})
  haystack = actual.to_s
  haystack = haystack.downcase if ignore_case
  needles = Array(expected).map(&:to_s)

  # Recorded rather than merely counted: which of five keywords went missing is what
  # makes the failure debuggable.
  missing = needles.reject do |needle|
    haystack.include?(ignore_case ? needle.downcase : needle)
  end

  metadata = result_metadata.merge(matcher_metadata(actual, expected)).merge(missing: missing)

  expect label || "includes expected text", result_metadata: metadata do
    needles.any? && missing.empty?
  end
end

#expect_matches(actual, pattern, label: nil, result_metadata: {}) ⇒ ExpectationResult

Asserts that a value matches a pattern. A String pattern is compiled, so a dataset row can carry one.

Examples:

expect_matches(task.parsed_response, /\A[A-Z]{2}-\d{4}\z/)

Parameters:

  • actual (Object) —

    the value produced by whatever is under test, read as text

  • pattern (Regexp, String) —

    the pattern to match

  • label (String, nil) (defaults to: nil) —

    replaces the expectation's description

  • result_metadata (Hash) (defaults to: {}) —

    merged into the recorded metadata

Returns:



89
90
91
92
93
94
95
96
 
# File 'lib/raif/evals/eval_sets/matchers.rb', line 89

def expect_matches(actual, pattern, label: nil, result_metadata: {})
  regexp = pattern.is_a?(Regexp) ? pattern : Regexp.new(pattern.to_s)
  metadata = result_metadata.merge(matcher_metadata(actual, regexp.source)).merge(pattern: regexp.inspect)

  expect label || "matches expected pattern", result_metadata: metadata do
    regexp.match?(actual.to_s)
  end
end

#expect_within(actual, expected, delta: nil, percent: nil, label: nil, result_metadata: {}) ⇒ ExpectationResult

Asserts that a number is close enough to ground truth. Mirrors RSpec's be_within(delta).of(expected), with percent: as the relative alternative.

A non-numeric actual fails rather than raises: a model that answered "about forty" when the eval asked for a number produced a wrong answer, not a broken eval. A non-numeric expected raises, because only the eval's author can have put it there.

Examples:

expect_within(task.parsed_response["total"], eval_case.expected["total"], percent: 1)

Parameters:

  • actual (Object) —

    the value produced by whatever is under test

  • expected (Numeric) —

    the ground-truth value

  • delta (Numeric, nil) (defaults to: nil) —

    the absolute tolerance; give this or percent:, not both

  • percent (Numeric, nil) (defaults to: nil) —

    the tolerance as a percentage of expected

  • label (String, nil) (defaults to: nil) —

    replaces the expectation's description

  • result_metadata (Hash) (defaults to: {}) —

    merged into the recorded metadata

Returns:

Raises:

  • (ArgumentError) —

    when expected is not numeric, or the tolerance is not exactly one of delta: and percent:



119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
 
# File 'lib/raif/evals/eval_sets/matchers.rb', line 119

def expect_within(actual, expected, delta: nil, percent: nil, label: nil, result_metadata: {})
  if delta.nil? == percent.nil?
    raise ArgumentError, "expect_within needs exactly one of delta: and percent:, and was given " \
      "#{delta.nil? ? "neither" : "both"}."
  end

  unless expected.is_a?(Numeric)
    raise ArgumentError, "expect_within was given #{expected.inspect} as the expected value; it must be numeric."
  end

  # A percentage of zero is zero, so a zero expected admits only an exact zero.
  tolerance = delta || (expected.abs * percent.to_f / 100.0)
  difference = (actual - expected).abs if actual.is_a?(Numeric)

  metadata = result_metadata
    .merge(matcher_metadata(actual, expected))
    .merge({ tolerance: tolerance.to_f, difference: difference&.to_f }.compact)

  description = delta ? "within #{delta} of expected" : "within #{percent}% of expected"

  expect label || description, result_metadata: metadata do
    !difference.nil? && difference <= tolerance
  end
end