Class: Raif::Evals::RunReport

Inherits:
Object
  • Object
show all
Defined in:
lib/raif/evals/run_report.rb

Overview

Renders one eval run's results file as a self-contained HTML page.

Raif::Evals::ComparisonReport answers "what moved between two runs". This answers "what happened in this one", which is the question a run with no baseline leaves you holding: the console output scrolls away, and the results JSON is the wrong shape to read by eye.

Input is the parsed results file. Keys are normalized to strings on the way in, so the hash Raif::Evals::Run holds in memory renders the same as the file it writes.

Constant Summary collapse

TEMPLATE_PATH = 
File.expand_path("run_report.html.erb", __dir__)
STYLESHEET_PATH = 
File.expand_path("report.css", __dir__)
FORMATS = 
["html"].freeze
COMPLETION_TEXT_KEYS =

What Raif::Evals::EvalResult strips under summary capture and keeps under full. Present in the file only in the second case, which is why the footer says which one it was.

["system_prompt", "messages", "response", "response_array", "response_tool_calls"].freeze

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(payload, label: nil) ⇒ RunReport

Returns a new instance of RunReport.



43
44
45
46
 
# File 'lib/raif/evals/run_report.rb', line 43

def initialize(payload, label: nil)
  @payload = stringify(payload)
  @label = label
end

Instance Attribute Details

#label ⇒ Object (readonly)

Returns the value of attribute label.



26
27
28
 
# File 'lib/raif/evals/run_report.rb', line 26

def label
  @label
end

#payload ⇒ Object (readonly)

Returns the value of attribute payload.



26
27
28
 
# File 'lib/raif/evals/run_report.rb', line 26

def payload
  @payload
end

Class Method Details

.format_duration(seconds) ⇒ Object

"45s", "12m 04s", "1h 03m". Shared with Raif::Evals::LiveReport, so the two pages agree.



29
30
31
32
33
34
35
36
37
38
39
40
41
 
# File 'lib/raif/evals/run_report.rb', line 29

def self.format_duration(seconds)
  seconds = seconds.to_i
  hours, remainder = seconds.divmod(3600)
  minutes, secs = remainder.divmod(60)

  if hours.positive?
    format("%dh %02dm", hours, minutes)
  elsif minutes.positive?
    format("%dm %02ds", minutes, secs)
  else
    "#{secs}s"
  end
end

Instance Method Details

#capture_mode ⇒ Object

"none" rather than "" for a run recorded before the setting existed: the report says what the file holds, and an empty cell says nothing.



98
99
100
101
 
# File 'lib/raif/evals/run_report.rb', line 98

def capture_mode
  mode = configuration["capture_model_completions"].to_s
  mode.empty? ? "none" : mode
end

#completion_text(completion) ⇒ Object 



178
179
180
181
182
183
 
# File 'lib/raif/evals/run_report.rb', line 178

def completion_text(completion)
  COMPLETION_TEXT_KEYS.filter_map do |key|
    value = completion[key]
    [key, value] if value.is_a?(String) ? !value.strip.empty? : !value.nil?
  end
end

#configuration ⇒ Object 



68
69
70
 
# File 'lib/raif/evals/run_report.rb', line 68

def configuration
  payload["configuration"] || {}
end

#dataset_rows ⇒ Object 



132
133
134
 
# File 'lib/raif/evals/run_report.rb', line 132

def dataset_rows
  configuration["datasets"] || []
end

#describe_code ⇒ Object 



185
186
187
188
189
190
 
# File 'lib/raif/evals/run_report.rb', line 185

def describe_code
  code = configuration["code"]
  return "unknown" if code.nil?

  "#{code["git_sha"].to_s[0, 12]}#{" (dirty)" if code["dirty"]}"
end

#elapsed ⇒ Object

"-" for a results file written before the elapsed time was recorded.



173
174
175
176
 
# File 'lib/raif/evals/run_report.rb', line 173

def elapsed
  seconds = payload["elapsed_seconds"]
  seconds.nil? ? "-" : self.class.format_duration(seconds)
end

#errored_cases(row) ⇒ Object 



226
227
228
 
# File 'lib/raif/evals/run_report.rb', line 226

def errored_cases(row)
  (row["per_case"] || []).select { |c| c["pass_rate"].nil? }
end

#errored_evals ⇒ Object 



148
149
150
 
# File 'lib/raif/evals/run_report.rb', line 148

def errored_evals
  summary["errored_evals"].to_i
end

#eval_set_class(results) ⇒ Object

An eval set whose only shortfall is errors is not a failing eval set: nothing it ran measured worse, so it reads as a warning rather than as red.



241
242
243
244
245
246
 
# File 'lib/raif/evals/run_report.rb', line 241

def eval_set_class(results)
  measured_results = results.reject { |result| result["errored"] }
  return "bad" unless measured_results.all? { |result| result["passed"] }

  measured_results.count == results.count ? "good" : "warn"
end

#eval_sets ⇒ Object

Eval set name => its results, in the order the results file holds them, which Raif::Evals::Run has already put back into definition order.



74
75
76
 
# File 'lib/raif/evals/run_report.rb', line 74

def eval_sets
  payload["results"] || {}
end

#failed_evals ⇒ Object 



152
153
154
 
# File 'lib/raif/evals/run_report.rb', line 152

def failed_evals
  total_evals - passed_evals - errored_evals
end

#failing_cases(row) ⇒ Object

Cases that measured something and did not pass every run. A case with no rate measured nothing, so it is reported as errored rather than as the weakest case in the eval.



222
223
224
 
# File 'lib/raif/evals/run_report.rb', line 222

def failing_cases(row)
  (row["per_case"] || []).reject { |c| c["pass_rate"].nil? || c["pass_rate"].to_f >= 1.0 }
end

#failures ⇒ Object

Every expectation that did not pass, flattened across the whole run and carrying enough context to find it again. This is the section a reader wants first, and reaching it by scrolling a full drill-down defeats the point of writing one.



106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
 
# File 'lib/raif/evals/run_report.rb', line 106

def failures
  eval_sets.flat_map do |eval_set_name, results|
    results.flat_map do |result|
      (result["expectation_results"] || [])
        .reject { |expectation| expectation["status"].to_s == "passed" }
        .map do |expectation|
          {
            "eval_set" => eval_set_name,
            "description" => result["description"],
            "case_id" => result["case_id"],
            "run_index" => result["run_index"],
            "expectation" => expectation
          }
        end
    end
  end
end

#format_cost(cost) ⇒ Object 



259
260
261
262
263
 
# File 'lib/raif/evals/run_report.rb', line 259

def format_cost(cost)
  return "-" if cost.nil?

  "$#{format("%.2f", cost.to_f)}"
end

#format_number(value) ⇒ Object 



265
266
267
 
# File 'lib/raif/evals/run_report.rb', line 265

def format_number(value)
  value.to_i.to_s.reverse.scan(/\d{1,3}/).join(",").reverse
end

#format_rate(rate) ⇒ Object

nil rather than 0.0 when every run of an eval or a case errored: Raif::Evals::Run leaves the rate unset there, and a zero would report an outage as a total quality failure.



202
203
204
205
206
 
# File 'lib/raif/evals/run_report.rb', line 202

def format_rate(rate)
  return "-" if rate.nil?

  format("%.2f", rate.to_f)
end

#gate_description(score) ⇒ Object

The bounds the score was actually gated on, in the shape Raif::Evals::ScoreResult states them: a max: gate is the supported form for a lower-is-better metric, and a score can carry both bounds.



251
252
253
254
255
256
257
 
# File 'lib/raif/evals/run_report.rb', line 251

def gate_description(score)
  bounds = []
  bounds << ">= #{score["min"]}" unless score["min"].nil?
  bounds << "<= #{score["max"]}" unless score["max"].nil?

  bounds.join(" and ")
end

#h(value) ⇒ Object

Model output reaches this page as expectation metadata and judge reasoning, so every interpolation in the template goes through here. A report is a file people forward.



271
272
273
 
# File 'lib/raif/evals/run_report.rb', line 271

def h(value)
  ERB::Util.html_escape(value.to_s)
end

#html ⇒ Object 



55
56
57
 
# File 'lib/raif/evals/run_report.rb', line 55

def html
  ERB.new(File.read(TEMPLATE_PATH), trim_mode: "-").result(binding)
end

#judge ⇒ Object

What actually graded, rather than what was configured: a run with no judge configured is graded by the model under test, and the configured key is null for it.



88
89
90
 
# File 'lib/raif/evals/run_report.rb', line 88

def judge
  configuration["judge_model_key"] || configuration["evals_default_llm_judge_model_key"]
end

#measured(row) ⇒ Object

What passed is out of: errored runs leave the denominator, so a row that ran 4 times, passed 3 and errored once is 3/3.



216
217
218
 
# File 'lib/raif/evals/run_report.rb', line 216

def measured(row)
  row["runs"].to_i - row["errored"].to_i
end

#model ⇒ Object 



82
83
84
 
# File 'lib/raif/evals/run_report.rb', line 82

def model
  configuration["default_llm_model_key"]
end

#pass_rate_rows ⇒ Object 



124
125
126
 
# File 'lib/raif/evals/run_report.rb', line 124

def pass_rate_rows
  summary["eval_pass_rates"] || []
end

#passed_evals ⇒ Object 



144
145
146
 
# File 'lib/raif/evals/run_report.rb', line 144

def passed_evals
  summary["passed_evals"].to_i
end

#percent_of_measured(count) ⇒ Object

Of the evals that measured something: errored ones leave the denominator, as they do in every pass rate Raif reports. nil when nothing was measured.



158
159
160
161
162
163
 
# File 'lib/raif/evals/run_report.rb', line 158

def percent_of_measured(count)
  measured = total_evals - errored_evals
  return if measured.zero?

  (count * 100.0 / measured).round
end

#percent_of_total(count) ⇒ Object

A width for the progress bar, as a share of every eval in the run.



166
167
168
169
170
 
# File 'lib/raif/evals/run_report.rb', line 166

def percent_of_total(count)
  return 0 if total_evals.zero?

  (count * 100.0 / total_evals).round(3)
end

#pretty(value) ⇒ Object 



275
276
277
278
279
 
# File 'lib/raif/evals/run_report.rb', line 275

def pretty(value)
  JSON.pretty_generate(value)
rescue JSON::GeneratorError, TypeError
  value.inspect
end

#rate_class(rate) ⇒ Object 



208
209
210
211
212
 
# File 'lib/raif/evals/run_report.rb', line 208

def rate_class(rate)
  return "warn" if rate.nil?

  rate.to_f >= 1.0 ? "good" : "bad"
end

#render(format = "html") ⇒ Object 



48
49
50
51
52
53
 
# File 'lib/raif/evals/run_report.rb', line 48

def render(format = "html")
  case format.to_s
  when "html" then html
  else raise ArgumentError, "Unsupported format: #{format}. Supported: #{FORMATS.join(", ")}"
  end
end

#repeats ⇒ Object 



92
93
94
 
# File 'lib/raif/evals/run_report.rb', line 92

def repeats
  configuration["repeats"] || 1
end

#result_status(result) ⇒ Object

An eval that raised produced no measurement, so it is neither a pass nor a fail. Raif keeps the three apart everywhere else, and a red FAIL on a provider timeout reads as a quality regression.



233
234
235
236
237
 
# File 'lib/raif/evals/run_report.rb', line 233

def result_status(result)
  return ["ERROR", "warn"] if result["errored"]

  result["passed"] ? ["PASS", "good"] : ["FAIL", "bad"]
end

#run_at ⇒ Object 



78
79
80
 
# File 'lib/raif/evals/run_report.rb', line 78

def run_at
  payload["run_at"].to_s.sub("T", " ")[0, 16]
end

#score_rows ⇒ Object 



128
129
130
 
# File 'lib/raif/evals/run_report.rb', line 128

def score_rows
  summary["score_summaries"] || []
end

#status_class(status) ⇒ Object 



192
193
194
195
196
197
198
 
# File 'lib/raif/evals/run_report.rb', line 192

def status_class(status)
  case status.to_s
  when "passed" then "good"
  when "error" then "warn"
  else "bad"
  end
end

#stylesheet ⇒ Object

Inlined rather than linked: the page is a single file that people forward.



60
61
62
 
# File 'lib/raif/evals/run_report.rb', line 60

def stylesheet
  File.read(STYLESHEET_PATH)
end

#summary ⇒ Object 



64
65
66
 
# File 'lib/raif/evals/run_report.rb', line 64

def summary
  payload["summary"] || {}
end

#total_cost ⇒ Object 



136
137
138
 
# File 'lib/raif/evals/run_report.rb', line 136

def total_cost
  summary["total_cost"].to_f
end

#total_evals ⇒ Object 



140
141
142
 
# File 'lib/raif/evals/run_report.rb', line 140

def total_evals
  summary["total_evals"].to_i
end