Class: Raif::Evals::Comparison

Inherits:
Object
  • Object
show all
Defined in:
lib/raif/evals/comparison.rb

Overview

Diffs two eval run payloads. Free of Rails and of any provider call, so it needs no database, API key, or spend.

Results are matched on eval id, case id, and expectation description. Per-case matching is what makes a dataset run diffable: a model can improve on average while getting worse on one input.

Constant Summary collapse

FAMILY_WISE_ALPHA =

Family-wise significance level for the regression gate: the chance the whole comparison reports at least one regression that was only noise. Per-row levels are this divided by the number of rows tested - see #significant_regressions.

0.05
MAX_GATE_ERROR_RATE =

The share of runs either side may lose to errors before the regression gate refuses to decide - see #error_rate_unreliable?.

0.05

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(baseline:, candidate:, baseline_label: nil, candidate_label: nil) ⇒ Comparison

Returns a new instance of Comparison.



23
24
25
26
27
28
 
# File 'lib/raif/evals/comparison.rb', line 23

def initialize(baseline:, candidate:, baseline_label: nil, candidate_label: nil)
  @baseline = baseline
  @candidate = candidate
  @baseline_label = baseline_label
  @candidate_label = candidate_label
end

Instance Attribute Details

#baseline ⇒ Object (readonly)

Returns the value of attribute baseline.



21
22
23
 
# File 'lib/raif/evals/comparison.rb', line 21

def baseline
  @baseline
end

#baseline_label ⇒ Object (readonly)

Returns the value of attribute baseline_label.



21
22
23
 
# File 'lib/raif/evals/comparison.rb', line 21

def baseline_label
  @baseline_label
end

#candidate ⇒ Object (readonly)

Returns the value of attribute candidate.



21
22
23
 
# File 'lib/raif/evals/comparison.rb', line 21

def candidate
  @candidate
end

#candidate_label ⇒ Object (readonly)

Returns the value of attribute candidate_label.



21
22
23
 
# File 'lib/raif/evals/comparison.rb', line 21

def candidate_label
  @candidate_label
end

Instance Method Details

#baseline_code ⇒ Object 



64
65
66
 
# File 'lib/raif/evals/comparison.rb', line 64

def baseline_code
  code(baseline)
end

#baseline_error_rate ⇒ Object 



97
98
99
 
# File 'lib/raif/evals/comparison.rb', line 97

def baseline_error_rate
  @baseline_error_rate ||= overall_error_rate(baseline_units)
end

#baseline_judge ⇒ Object 



30
31
32
 
# File 'lib/raif/evals/comparison.rb', line 30

def baseline_judge
  judge_model(baseline)
end

#candidate_code ⇒ Object 



68
69
70
 
# File 'lib/raif/evals/comparison.rb', line 68

def candidate_code
  code(candidate)
end

#candidate_error_rate ⇒ Object 



101
102
103
 
# File 'lib/raif/evals/comparison.rb', line 101

def candidate_error_rate
  @candidate_error_rate ||= overall_error_rate(candidate_units)
end

#candidate_judge ⇒ Object 



34
35
36
 
# File 'lib/raif/evals/comparison.rb', line 34

def candidate_judge
  judge_model(candidate)
end

#candidate_regressions(threshold) ⇒ Object

The rows big enough to gate on, before asking whether they are distinguishable from noise. Everything below derives from this set, so the evidence bar is only ever applied to rows that already cleared the size bar.



144
145
146
147
148
 
# File 'lib/raif/evals/comparison.rb', line 144

def candidate_regressions(threshold)
  return [] if threshold.nil?

  regressions.select { |row| row[:magnitude].nil? || row[:magnitude] > threshold.to_f }
end

#dataset_differences ⇒ Object

Datasets whose contents are not identical across the two runs. Cases are joined on case id, so an edited case reads as the model behaving differently on the same input.

Reported rather than refused, unlike a judge mismatch: comparing a widened dataset against its predecessor is legitimate, as long as the reader knows that is what they are seeing.



56
57
58
 
# File 'lib/raif/evals/comparison.rb', line 56

def dataset_differences
  @dataset_differences ||= build_dataset_differences
end

#dataset_mismatch? ⇒ Boolean

Returns:

  • (Boolean) 


60
61
62
 
# File 'lib/raif/evals/comparison.rb', line 60

def dataset_mismatch?
  dataset_differences.any?
end

#dataset_provenance? ⇒ Boolean

Whether both runs recorded what their datasets held. A run written before provenance existed has no datasets key at all, which is not the same as a run that used no dataset, so the fingerprint check says nothing rather than claiming a match it cannot support.

Returns:

  • (Boolean) 


47
48
49
 
# File 'lib/raif/evals/comparison.rb', line 47

def dataset_provenance?
  [baseline, candidate].all? { |payload| (payload["configuration"] || {}).key?("datasets") }
end

#error_moves ⇒ Object

How often each side raised instead of producing a measurement. Reported on its own rather than through the pass rates, which exclude errors: a run that got worse and a run that got flakier need different responses.



93
94
95
 
# File 'lib/raif/evals/comparison.rb', line 93

def error_moves
  @error_moves ||= build_error_moves
end

#error_rate_unreliable?(max_error_rate: MAX_GATE_ERROR_RATE) ⇒ Boolean

Errors leave the pass-rate denominator, but that leaves selection bias: if the runs that errored were not a random sample - the long inputs are the ones that time out - the surviving denominator is biased, and the bias scales with how many were lost. Past this rate callers refuse to gate rather than gate badly.

Parameters:

  • max_error_rate (Numeric) (defaults to: MAX_GATE_ERROR_RATE) —

    the fraction of runs allowed to error on either side.

Returns:

  • (Boolean) 


111
112
113
 
# File 'lib/raif/evals/comparison.rb', line 111

def error_rate_unreliable?(max_error_rate: MAX_GATE_ERROR_RATE)
  [baseline_error_rate, candidate_error_rate].max > max_error_rate.to_f
end

#fixed ⇒ Object 



76
77
78
 
# File 'lib/raif/evals/comparison.rb', line 76

def fixed
  eval_moves[:improved]
end

#insufficient_evidence?(threshold, alpha: FAMILY_WISE_ALPHA) ⇒ Boolean

True when the gate was asked a question it has no evidence to answer: something moved past the threshold, nothing that moved could be tested. Distinct from "nothing regressed", and callers report it rather than exit 0 on it.

Returns:

  • (Boolean) 


185
186
187
188
189
190
 
# File 'lib/raif/evals/comparison.rb', line 185

def insufficient_evidence?(threshold, alpha: FAMILY_WISE_ALPHA)
  return false if threshold.nil? || evidence_waived?(alpha)

  candidates = candidate_regressions(threshold)
  candidates.any? && candidates.all? { |row| row[:p_value].nil? }
end

#judge_mismatch? ⇒ Boolean

Scores from two different judges measure two different things, so callers refuse to compare rather than warn.

Returns:

  • (Boolean) 


40
41
42
 
# File 'lib/raif/evals/comparison.rb', line 40

def judge_mismatch?
  baseline_judge != candidate_judge
end

#max_regression ⇒ Object

Excludes unbounded regressions, whose magnitude would be Float::INFINITY and cannot be exported to JSON. Ask #regressed? for the gate decision, which accounts for both.



137
138
139
 
# File 'lib/raif/evals/comparison.rb', line 137

def max_regression
  regressions.filter_map { |row| row[:magnitude] }.max || 0.0
end

#new_failures ⇒ Object 



72
73
74
 
# File 'lib/raif/evals/comparison.rb', line 72

def new_failures
  eval_moves[:regressed]
end

#not_comparable ⇒ Object

Cases and expectations present on only one side, plus the ones a side errored out of entirely. Surfaced rather than dropped: a silently omitted case looks like agreement.



86
87
88
 
# File 'lib/raif/evals/comparison.rb', line 86

def not_comparable
  @not_comparable ||= build_not_comparable
end

#regressed?(threshold, alpha: FAMILY_WISE_ALPHA) ⇒ Boolean

Parameters:

  • threshold (Numeric, nil) —

    the effect size gate, as a fraction of baseline. nil never fails.

  • alpha (Numeric) (defaults to: FAMILY_WISE_ALPHA) —

    family-wise significance level. 1 or greater waives the evidence requirement, gating on point estimates alone - all a three-case dataset can support, since three matched pairs cannot reach any conventional level however large the drop.

Returns:

  • (Boolean) 


176
177
178
179
180
 
# File 'lib/raif/evals/comparison.rb', line 176

def regressed?(threshold, alpha: FAMILY_WISE_ALPHA)
  return false if threshold.nil?

  significant_regressions(threshold, alpha: alpha).any?
end

#regressions ⇒ Object

Every regression either run shows, each with two independent halves: how big it is (#magnitude) and how sure we can be it is not noise (#p_value). #regressed? demands both, because either alone gates badly - a size threshold fires on run-to-run variation, and significance with no floor on size fails a run over a precisely measured rounding error.

Magnitudes are relative to the baseline, since a pass rate, a rubric score, and a latency are not in the same units. Rows come from every comparable eval, not just new_failures: an eval that fixed one expectation and broke another can come out ahead on its rate and still be a trade.



124
125
126
127
 
# File 'lib/raif/evals/comparison.rb', line 124

def regressions
  # An unbounded row (nil magnitude) sorts first: it is the worst thing in the list.
  @regressions ||= (pass_rate_regressions + score_regressions).sort_by { |row| -(row[:magnitude] || Float::INFINITY) }
end

#score_moves ⇒ Object 



80
81
82
 
# File 'lib/raif/evals/comparison.rb', line 80

def score_moves
  @score_moves ||= build_score_moves
end

#significant_regressions(threshold, alpha: FAMILY_WISE_ALPHA) ⇒ Object

Candidates that cleared both bars. The per-row level is the family-wise alpha divided by the number of candidates (Bonferroni): the gate fails if ANY row is significant, so testing each at 0.05 would fail one run in three of a 20-row suite on noise alone. Dividing by the candidate count rather than by every row in the report keeps the correction as loose as it can honestly be.



162
163
164
165
166
167
168
169
 
# File 'lib/raif/evals/comparison.rb', line 162

def significant_regressions(threshold, alpha: FAMILY_WISE_ALPHA)
  candidates = candidate_regressions(threshold)
  return candidates if evidence_waived?(alpha)
  return [] if candidates.empty?

  per_row_alpha = alpha.to_f / candidates.count
  candidates.select { |row| row[:p_value] && row[:p_value] <= per_row_alpha }
end

#to_h ⇒ Object 



192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
 
# File 'lib/raif/evals/comparison.rb', line 192

def to_h
  @to_h ||= {
    baseline: side_summary(baseline, baseline_label, baseline_units),
    candidate: side_summary(candidate, candidate_label, candidate_units),
    judge_mismatch: judge_mismatch?,
    dataset_mismatch: dataset_mismatch?,
    dataset_differences: dataset_differences,
    new_failures: new_failures,
    fixed: fixed,
    score_moves: score_moves,
    error_moves: error_moves,
    not_comparable: not_comparable,
    regressions: regressions
  }
end

#unbounded_regressions ⇒ Object

Rows with no fraction to take because the baseline was zero. "0 errors became 3" is still a regression, so #regressed? treats it as exceeding any threshold.



131
132
133
 
# File 'lib/raif/evals/comparison.rb', line 131

def unbounded_regressions
  regressions.select { |row| row[:magnitude].nil? }
end

#unverifiable_regressions(threshold) ⇒ Object

Candidates whose move cannot be tested at all: a score on an eval with no dataset, where repeat 3 of one run is not the counterpart of repeat 3 of the other. Reported rather than silently passed, so a run with a real regression does not exit 0 looking green.



153
154
155
 
# File 'lib/raif/evals/comparison.rb', line 153

def unverifiable_regressions(threshold)
  candidate_regressions(threshold).select { |row| row[:p_value].nil? }
end