Class: Raif::Evals::RunReport

Inherits:
Object
  • Object
show all
Defined in:
lib/raif/evals/run_report.rb

Overview

Renders one eval run's results file as a self-contained HTML page.

Raif::Evals::ComparisonReport answers "what moved between two runs". This answers "what happened in this one", which is the question a run with no baseline leaves you holding: the console output scrolls away, and the results JSON is the wrong shape to read by eye.

Input is the parsed results file. Keys are normalized to strings on the way in, so the hash Raif::Evals::Run holds in memory renders the same as the file it writes.

Constant Summary collapse

TEMPLATE_PATH = 
File.expand_path("run_report.html.erb", __dir__)
FORMATS = 
["html"].freeze
COMPLETION_TEXT_KEYS =

What Raif::Evals::EvalResult strips under summary capture and keeps under full. Present in the file only in the second case, which is why the footer says which one it was.

["system_prompt", "messages", "response", "response_array", "response_tool_calls"].freeze

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(payload, label: nil) ⇒ RunReport

Returns a new instance of RunReport.



27
28
29
30
 
# File 'lib/raif/evals/run_report.rb', line 27

def initialize(payload, label: nil)
  @payload = stringify(payload)
  @label = label
end

Instance Attribute Details

#label ⇒ Object (readonly)

Returns the value of attribute label.



25
26
27
 
# File 'lib/raif/evals/run_report.rb', line 25

def label
  @label
end

#payload ⇒ Object (readonly)

Returns the value of attribute payload.



25
26
27
 
# File 'lib/raif/evals/run_report.rb', line 25

def payload
  @payload
end

Instance Method Details

#capture_mode ⇒ Object

"none" rather than "" for a run recorded before the setting existed: the report says what the file holds, and an empty cell says nothing.



77
78
79
80
 
# File 'lib/raif/evals/run_report.rb', line 77

def capture_mode
  mode = configuration["capture_model_completions"].to_s
  mode.empty? ? "none" : mode
end

#completion_text(completion) ⇒ Object 



118
119
120
121
122
123
 
# File 'lib/raif/evals/run_report.rb', line 118

def completion_text(completion)
  COMPLETION_TEXT_KEYS.filter_map do |key|
    value = completion[key]
    [key, value] if value.is_a?(String) ? !value.strip.empty? : !value.nil?
  end
end

#configuration ⇒ Object 



47
48
49
 
# File 'lib/raif/evals/run_report.rb', line 47

def configuration
  payload["configuration"] || {}
end

#dataset_rows ⇒ Object 



110
111
112
 
# File 'lib/raif/evals/run_report.rb', line 110

def dataset_rows
  configuration["datasets"] || []
end

#describe_code ⇒ Object 



125
126
127
128
129
130
 
# File 'lib/raif/evals/run_report.rb', line 125

def describe_code
  code = configuration["code"]
  return "unknown" if code.nil?

  "#{code["git_sha"].to_s[0, 12]}#{" (dirty)" if code["dirty"]}"
end

#errored_cases(row) ⇒ Object 



166
167
168
 
# File 'lib/raif/evals/run_report.rb', line 166

def errored_cases(row)
  (row["per_case"] || []).select { |c| c["pass_rate"].nil? }
end

#eval_set_class(results) ⇒ Object

An eval set whose only shortfall is errors is not a failing eval set: nothing it ran measured worse, so it reads as a warning rather than as red.



181
182
183
184
185
186
 
# File 'lib/raif/evals/run_report.rb', line 181

def eval_set_class(results)
  measured_results = results.reject { |result| result["errored"] }
  return "bad" unless measured_results.all? { |result| result["passed"] }

  measured_results.count == results.count ? "good" : "warn"
end

#eval_sets ⇒ Object

Eval set name => its results, in the order the results file holds them, which Raif::Evals::Run has already put back into definition order.



53
54
55
 
# File 'lib/raif/evals/run_report.rb', line 53

def eval_sets
  payload["results"] || {}
end

#failing_cases(row) ⇒ Object

Cases that measured something and did not pass every run. A case with no rate measured nothing, so it is reported as errored rather than as the weakest case in the eval.



162
163
164
 
# File 'lib/raif/evals/run_report.rb', line 162

def failing_cases(row)
  (row["per_case"] || []).reject { |c| c["pass_rate"].nil? || c["pass_rate"].to_f >= 1.0 }
end

#failures ⇒ Object

Every expectation that did not pass, flattened across the whole run and carrying enough context to find it again. This is the section a reader wants first, and reaching it by scrolling a full drill-down defeats the point of writing one.



85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
 
# File 'lib/raif/evals/run_report.rb', line 85

def failures
  eval_sets.flat_map do |eval_set_name, results|
    results.flat_map do |result|
      (result["expectation_results"] || [])
        .reject { |expectation| expectation["status"].to_s == "passed" }
        .map do |expectation|
          {
            "eval_set" => eval_set_name,
            "description" => result["description"],
            "case_id" => result["case_id"],
            "expectation" => expectation
          }
        end
    end
  end
end

#format_cost(cost) ⇒ Object 



199
200
201
202
203
 
# File 'lib/raif/evals/run_report.rb', line 199

def format_cost(cost)
  return "-" if cost.nil?

  "$#{format("%.2f", cost.to_f)}"
end

#format_number(value) ⇒ Object 



205
206
207
 
# File 'lib/raif/evals/run_report.rb', line 205

def format_number(value)
  value.to_i.to_s.reverse.scan(/\d{1,3}/).join(",").reverse
end

#format_rate(rate) ⇒ Object

nil rather than 0.0 when every run of an eval or a case errored: Raif::Evals::Run leaves the rate unset there, and a zero would report an outage as a total quality failure.



142
143
144
145
146
 
# File 'lib/raif/evals/run_report.rb', line 142

def format_rate(rate)
  return "-" if rate.nil?

  format("%.2f", rate.to_f)
end

#gate_description(score) ⇒ Object

The bounds the score was actually gated on, in the shape Raif::Evals::ScoreResult states them: a max: gate is the supported form for a lower-is-better metric, and a score can carry both bounds.



191
192
193
194
195
196
197
 
# File 'lib/raif/evals/run_report.rb', line 191

def gate_description(score)
  bounds = []
  bounds << ">= #{score["min"]}" unless score["min"].nil?
  bounds << "<= #{score["max"]}" unless score["max"].nil?

  bounds.join(" and ")
end

#h(value) ⇒ Object

Model output reaches this page as expectation metadata and judge reasoning, so every interpolation in the template goes through here. A report is a file people forward.



211
212
213
 
# File 'lib/raif/evals/run_report.rb', line 211

def h(value)
  ERB::Util.html_escape(value.to_s)
end

#html ⇒ Object 



39
40
41
 
# File 'lib/raif/evals/run_report.rb', line 39

def html
  ERB.new(File.read(TEMPLATE_PATH), trim_mode: "-").result(binding)
end

#judge ⇒ Object

What actually graded, rather than what was configured: a run with no judge configured is graded by the model under test, and the configured key is null for it.



67
68
69
 
# File 'lib/raif/evals/run_report.rb', line 67

def judge
  configuration["judge_model_key"] || configuration["evals_default_llm_judge_model_key"]
end

#measured(row) ⇒ Object

What passed is out of: errored runs leave the denominator, so a row that ran 4 times, passed 3 and errored once is 3/3.



156
157
158
 
# File 'lib/raif/evals/run_report.rb', line 156

def measured(row)
  row["runs"].to_i - row["errored"].to_i
end

#model ⇒ Object 



61
62
63
 
# File 'lib/raif/evals/run_report.rb', line 61

def model
  configuration["default_llm_model_key"]
end

#pass_rate_rows ⇒ Object 



102
103
104
 
# File 'lib/raif/evals/run_report.rb', line 102

def pass_rate_rows
  summary["eval_pass_rates"] || []
end

#pretty(value) ⇒ Object 



215
216
217
218
219
 
# File 'lib/raif/evals/run_report.rb', line 215

def pretty(value)
  JSON.pretty_generate(value)
rescue JSON::GeneratorError, TypeError
  value.inspect
end

#rate_class(rate) ⇒ Object 



148
149
150
151
152
 
# File 'lib/raif/evals/run_report.rb', line 148

def rate_class(rate)
  return "warn" if rate.nil?

  rate.to_f >= 1.0 ? "good" : "bad"
end

#render(format = "html") ⇒ Object 



32
33
34
35
36
37
 
# File 'lib/raif/evals/run_report.rb', line 32

def render(format = "html")
  case format.to_s
  when "html" then html
  else raise ArgumentError, "Unsupported format: #{format}. Supported: #{FORMATS.join(", ")}"
  end
end

#repeats ⇒ Object 



71
72
73
 
# File 'lib/raif/evals/run_report.rb', line 71

def repeats
  configuration["repeats"] || 1
end

#result_status(result) ⇒ Object

An eval that raised produced no measurement, so it is neither a pass nor a fail. Raif keeps the three apart everywhere else, and a red FAIL on a provider timeout reads as a quality regression.



173
174
175
176
177
 
# File 'lib/raif/evals/run_report.rb', line 173

def result_status(result)
  return ["ERROR", "warn"] if result["errored"]

  result["passed"] ? ["PASS", "good"] : ["FAIL", "bad"]
end

#run_at ⇒ Object 



57
58
59
 
# File 'lib/raif/evals/run_report.rb', line 57

def run_at
  payload["run_at"].to_s.sub("T", " ")[0, 16]
end

#score_rows ⇒ Object 



106
107
108
 
# File 'lib/raif/evals/run_report.rb', line 106

def score_rows
  summary["score_summaries"] || []
end

#status_class(status) ⇒ Object 



132
133
134
135
136
137
138
 
# File 'lib/raif/evals/run_report.rb', line 132

def status_class(status)
  case status.to_s
  when "passed" then "good"
  when "error" then "warn"
  else "bad"
  end
end

#summary ⇒ Object 



43
44
45
 
# File 'lib/raif/evals/run_report.rb', line 43

def summary
  payload["summary"] || {}
end

#total_cost ⇒ Object 



114
115
116
 
# File 'lib/raif/evals/run_report.rb', line 114

def total_cost
  summary["total_cost"].to_f
end