Class: Raif::Evals::LiveReport

Inherits:
Object
  • Object
show all
Defined in:
lib/raif/evals/live_report.rb

Overview

A self-contained HTML page that describes a run while it continues. It is rewritten as each execution starts and finishes, reloads itself in the browser, and is replaced by the full Raif::Evals::RunReport at the same path when the run completes.

Ruby writes the page, rather than JavaScript on the page reading the run log, because a page opened from disk cannot fetch a neighbouring file. The run log stays the source of truth: every rewrite is rendered from a snapshot of it, and this class adds only what the log cannot know - which executions are in flight and how long the finished ones took.

Each rewrite goes to a temporary file that is then renamed over the page, so a reload sees the previous page or the next one and never half of either.

Constant Summary collapse

TEMPLATE_PATH = 
File.expand_path("live_report.html.erb", __dir__)
REFRESH_SECONDS = 
30
MIN_TIMED_FOR_ESTIMATE =

Below this many timed executions an average says more about one slow case than about the run.

3

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(path:, run_log:, configuration:, concurrency:, output:) ⇒ LiveReport

Returns a new instance of LiveReport.



30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
 
# File 'lib/raif/evals/live_report.rb', line 30

def initialize(path:, run_log:, configuration:, concurrency:, output:)
  @path = Pathname.new(path.to_s)
  @run_log = run_log
  @configuration = configuration
  @concurrency = concurrency
  @output = output
  @running = {}
  @finish_order = []
  @durations = []
  @stopped = nil
  @disabled = false
  @discarded = false
  # Workers start and finish executions concurrently. One lock around render-and-write keeps
  # two rewrites from racing on the temporary file, and keeps the running list consistent.
  @mutex = Mutex.new
end

Instance Attribute Details

#path ⇒ Object (readonly)

Returns the value of attribute path.



28
29
30
 
# File 'lib/raif/evals/live_report.rb', line 28

def path
  @path
end

Instance Method Details

#complete!(&render_report) ⇒ Object

The block renders the full run report, which takes over the page's path. Rendered inside the guard in #publish, since the results file is already written and a report that fails to render is no reason to fail the run. Returns whether the report was written.



91
92
93
 
# File 'lib/raif/evals/live_report.rb', line 91

def complete!(&render_report)
  @mutex.synchronize { publish(final: true, &render_report) }
end

#completed ⇒ Object 



165
166
167
 
# File 'lib/raif/evals/live_report.rb', line 165

def completed
  results.count
end

#completed_rows ⇒ Object

Newest first. Results this invocation finished are ordered by when they finished; results a resume carried forward have no finish time here, so they follow in the order the log holds.



257
258
259
260
261
262
263
264
 
# File 'lib/raif/evals/live_report.rb', line 257

def completed_rows
  order = @finish_order.each_with_index.to_h

  results.each_with_index.sort_by do |(_eval_set, result), index|
    position = order[result_key(result)]
    position ? [0, -position] : [1, index]
  end.map(&:first)
end

#concurrency ⇒ Object 



150
151
152
 
# File 'lib/raif/evals/live_report.rb', line 150

def concurrency
  @concurrency
end

#counts ⇒ Object 



179
180
181
 
# File 'lib/raif/evals/live_report.rb', line 179

def counts
  @counts ||= tally(results.map { |_eval_set, result| result })
end

#describe_code ⇒ Object 



154
155
156
157
158
159
 
# File 'lib/raif/evals/live_report.rb', line 154

def describe_code
  code = @configuration[:code]
  return "unknown" if code.nil?

  "#{code[:git_sha].to_s[0, 12]}#{" (dirty)" if code[:dirty]}"
end

#discard! ⇒ Object

For a run that stopped before it recorded anything, whose run log is deleted with it.



96
97
98
99
100
101
 
# File 'lib/raif/evals/live_report.rb', line 96

def discard!
  @mutex.synchronize do
    @discarded = true
    FileUtils.rm_f(path)
  end
end

#display_path ⇒ Object 



114
115
116
117
118
 
# File 'lib/raif/evals/live_report.rb', line 114

def display_path
  path.relative_path_from(Rails.root).to_s
rescue StandardError
  path.to_s
end

#elapsed ⇒ Object

Across every invocation of a resumed run, so the page agrees with the report that replaces it.



192
193
194
 
# File 'lib/raif/evals/live_report.rb', line 192

def elapsed
  format_duration(@run_log.elapsed_seconds)
end

#eval_set_rows ⇒ Object

One row per eval set, in plan order. Expected comes from the plan, so a set with nothing finished yet still shows what it owes.



219
220
221
222
223
224
225
226
227
 
# File 'lib/raif/evals/live_report.rb', line 219

def eval_set_rows
  expected_by_set = @snapshot[:plan].keys.group_by { |eval_id, _case_id, _run_index| eval_id.split("#").first }
  names = expected_by_set.keys | @snapshot[:results].keys.map(&:to_s)

  names.map do |name|
    set_results = @snapshot[:results][name] || @snapshot[:results][name.to_sym] || []
    { name: name, expected: expected_by_set.fetch(name, []).size, completed: set_results.size }.merge(tally(set_results))
  end
end

#execution_finished(execution) ⇒ Object

Called whether or not the execution recorded a result, so a failure cannot leave it listed as running.



70
71
72
73
74
75
76
77
 
# File 'lib/raif/evals/live_report.rb', line 70

def execution_finished(execution)
  synchronized_publish do
    key = key_for(execution)
    entry = @running.delete(key)
    @durations << (monotonic_now - entry[:started]) if entry
    @finish_order << key
  end
end

#execution_started(execution, eval_set_name:) ⇒ Object 



55
56
57
58
59
60
61
62
63
64
65
66
 
# File 'lib/raif/evals/live_report.rb', line 55

def execution_started(execution, eval_set_name:)
  synchronized_publish do
    @running[key_for(execution)] = {
      eval_set: eval_set_name,
      description: execution.eval_definition.description,
      case_id: execution.eval_case&.id,
      run_index: execution.run_index,
      started_at: Time.current,
      started: monotonic_now
    }
  end
end

#expected ⇒ Object 



161
162
163
 
# File 'lib/raif/evals/live_report.rb', line 161

def expected
  @snapshot[:plan].size
end

#failures ⇒ Object

Every expectation that did not pass, in the order the results were recorded.



238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
 
# File 'lib/raif/evals/live_report.rb', line 238

def failures
  results.flat_map do |eval_set_name, result|
    Array(result[:expectation_results]).each_with_index.filter_map do |expectation, index|
      next if expectation[:status].to_s == "passed"

      {
        eval_set: eval_set_name,
        description: result[:description],
        case_id: result[:case_id],
        run_index: result[:run_index],
        expectation: expectation,
        key: "failure:#{result_key(result).join("|")}|#{index}"
      }
    end
  end
end

#format_cost(cost) ⇒ Object 



280
281
282
283
 
# File 'lib/raif/evals/live_report.rb', line 280

def format_cost(cost)
  value = cost.to_f
  value >= 1 ? format("$%.2f", value) : format("$%.4f", value)
end

#h(value) ⇒ Object

Model output reaches this page as expectation metadata and judge reasoning.



291
292
293
 
# File 'lib/raif/evals/live_report.rb', line 291

def h(value)
  ERB::Util.html_escape(value.to_s)
end

#judge ⇒ Object 



146
147
148
 
# File 'lib/raif/evals/live_report.rb', line 146

def judge
  @configuration[:judge_model_key] || @configuration[:evals_default_llm_judge_model_key]
end

#judge_cost ⇒ Object 



187
188
189
 
# File 'lib/raif/evals/live_report.rb', line 187

def judge_cost
  results.sum { |_eval_set, result| result.dig(:judge_usage, :total_cost).to_f }
end

#model ⇒ Object 



142
143
144
 
# File 'lib/raif/evals/live_report.rb', line 142

def model
  @configuration[:default_llm_model_key]
end

#open_in_browser ⇒ Object

Hands the page to the platform's opener and does not wait for it. Skipped when the page was never written, and a failed launch costs one warning, never the run.



105
106
107
108
109
110
111
112
 
# File 'lib/raif/evals/live_report.rb', line 105

def open_in_browser
  return if @disabled || !path.exist?

  pid = Process.spawn(*browser_command, path.to_s, out: File::NULL, err: File::NULL)
  Process.detach(pid)
rescue SystemCallError => e
  @output.puts Raif::Utils::Colors.yellow("\nCould not open the live report in a browser: #{e.message}")
end

#outstanding ⇒ Object 



169
170
171
 
# File 'lib/raif/evals/live_report.rb', line 169

def outstanding
  @snapshot[:outstanding].count
end

#percent_complete ⇒ Object 



173
174
175
176
177
 
# File 'lib/raif/evals/live_report.rb', line 173

def percent_complete
  return 0 if expected.zero?

  ((completed.to_f / expected) * 100).floor
end

#percent_of_measured(count) ⇒ Object

Of the executions that measured something: errored ones leave the denominator, as they do in every pass rate Raif reports.



198
199
200
201
202
203
 
# File 'lib/raif/evals/live_report.rb', line 198

def percent_of_measured(count)
  measured = completed - counts[:errored]
  return if measured.zero?

  (count * 100.0 / measured).round
end

#pretty(value) ⇒ Object 



295
296
297
298
299
 
# File 'lib/raif/evals/live_report.rb', line 295

def pretty(value)
  JSON.pretty_generate(value)
rescue JSON::GeneratorError, TypeError
  value.inspect
end

#refresh_seconds ⇒ Object

Everything below is read by the template, against the snapshot taken for one rewrite.



122
123
124
 
# File 'lib/raif/evals/live_report.rb', line 122

def refresh_seconds
  REFRESH_SECONDS
end

#rendered_at ⇒ Object 



134
135
136
 
# File 'lib/raif/evals/live_report.rb', line 134

def rendered_at
  @rendered_at.strftime("%Y-%m-%d %H:%M:%S")
end

#result_status(result) ⇒ Object 



266
267
268
269
270
 
# File 'lib/raif/evals/live_report.rb', line 266

def result_status(result)
  return ["ERROR", "warn"] if result[:errored]

  result[:passed] ? ["PASS", "good"] : ["FAIL", "bad"]
end

#run_at ⇒ Object 



138
139
140
 
# File 'lib/raif/evals/live_report.rb', line 138

def run_at
  @run_log.run_at.to_s.sub("T", " ")[0, 16]
end

#running? ⇒ Boolean

Returns:

  • (Boolean) 


126
127
128
 
# File 'lib/raif/evals/live_report.rb', line 126

def running?
  @stopped.nil?
end

#running_for(entry) ⇒ Object 



233
234
235
 
# File 'lib/raif/evals/live_report.rb', line 233

def running_for(entry)
  format_duration(monotonic_now - entry[:started])
end

#running_rows ⇒ Object 



229
230
231
 
# File 'lib/raif/evals/live_report.rb', line 229

def running_rows
  @running.values.sort_by { |entry| entry[:started] }
end

#start!(executions:) ⇒ Object

Parameters:

  • executions (Integer) —

    how many executions this invocation will run. The estimate of the time left is over these, since a resume narrowed to one file will not run the rest of the plan.



49
50
51
52
53
 
# File 'lib/raif/evals/live_report.rb', line 49

def start!(executions:)
  synchronized_publish do
    @executions = executions
  end
end

#status_class(status) ⇒ Object 



272
273
274
275
276
277
278
 
# File 'lib/raif/evals/live_report.rb', line 272

def status_class(status)
  case status.to_s
  when "passed" then "good"
  when "error" then "warn"
  else "bad"
  end
end

#stop!(reason:, resume_command: nil) ⇒ Object

The last rewrite of a run that ended without completing its plan. It carries no reload tag, so the page stops refreshing a run that is no longer there.



81
82
83
84
85
86
 
# File 'lib/raif/evals/live_report.rb', line 81

def stop!(reason:, resume_command: nil)
  synchronized_publish(final: true) do
    @running.clear
    @stopped = { reason: reason, resume_command: resume_command }
  end
end

#stopped ⇒ Object 



130
131
132
 
# File 'lib/raif/evals/live_report.rb', line 130

def stopped
  @stopped
end

#stylesheet ⇒ Object

Read once: the page is rendered on every execution start and finish.



286
287
288
 
# File 'lib/raif/evals/live_report.rb', line 286

def stylesheet
  @stylesheet ||= File.read(RunReport::STYLESHEET_PATH)
end

#time_left ⇒ Object

Average duration of this invocation's finished executions, times what it has left, spread over the workers. Results a resume carried forward have no timing, so they do not count.



207
208
209
210
211
212
213
214
215
 
# File 'lib/raif/evals/live_report.rb', line 207

def time_left
  return "-" if @durations.size < MIN_TIMED_FOR_ESTIMATE

  remaining = [@executions.to_i - @finish_order.size, 0].max
  return "-" if remaining.zero?

  average = @durations.sum / @durations.size
  "~#{format_duration(average * remaining / [@concurrency, remaining].min)}"
end

#total_cost ⇒ Object 



183
184
185
 
# File 'lib/raif/evals/live_report.rb', line 183

def total_cost
  results.sum { |_eval_set, result| result.dig(:usage, :total_cost).to_f }
end