Class: Raif::Evals::EvalResult

Inherits:
Object
  • Object
show all
Defined in:
lib/raif/evals/eval_result.rb

Overview

The outcome of one execution of an eval block, not the eval block itself. The block and its description live on an EvalDefinition; running one against one EvalCase produces one of these.

Constant Summary collapse

COMPLETION_TEXT_KEYS =

The text :summary capture drops. Tokens and cost survive in every mode, so the usage totals never depend on capture mode.

[:system_prompt, :messages, :response, :response_array, :response_tool_calls].freeze
EMPTY_USAGE = 
{
  model_completions: 0,
  prompt_tokens: 0,
  completion_tokens: 0,
  total_tokens: 0,
  total_cost: 0.0
}.freeze

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(description:, eval_id: nil, run_index: nil, eval_index: nil, case_id: nil) ⇒ EvalResult

eval_id identifies the eval block that produced this result and survives the file being edited around it; with case_id, the dataset input, it is the key evals:compare and --resume match on. eval_index is that eval's position, which orders results within one run.



27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
 
# File 'lib/raif/evals/eval_result.rb', line 27

def initialize(description:, eval_id: nil, run_index: nil, eval_index: nil, case_id: nil)
  @description = description
  @eval_id = eval_id
  @run_index = run_index
  @eval_index = eval_index
  @case_id = case_id
  @expectation_results = []
  @model_completions = []
  @scores = []
  @usage = EMPTY_USAGE.dup
  @overhead_usage = EMPTY_USAGE.dup
  @judge_usage = EMPTY_USAGE.dup
  # Seeded from the configured mode, so a result that never reaches
  # #record_model_completions - one built directly by a spec or a host app's own helper -
  # omits the key under :none like its siblings instead of carrying an empty array.
  @model_completions_captured = capture_mode_records_completions?
end

Instance Attribute Details

#case_id ⇒ Object (readonly)

Returns the value of attribute case_id.



21
22
23
 
# File 'lib/raif/evals/eval_result.rb', line 21

def case_id
  @case_id
end

#description ⇒ Object (readonly)

Returns the value of attribute description.



21
22
23
 
# File 'lib/raif/evals/eval_result.rb', line 21

def description
  @description
end

#eval_id ⇒ Object (readonly)

Returns the value of attribute eval_id.



21
22
23
 
# File 'lib/raif/evals/eval_result.rb', line 21

def eval_id
  @eval_id
end

#eval_index ⇒ Object (readonly)

Returns the value of attribute eval_index.



21
22
23
 
# File 'lib/raif/evals/eval_result.rb', line 21

def eval_index
  @eval_index
end

#expectation_results ⇒ Object (readonly)

Returns the value of attribute expectation_results.



21
22
23
 
# File 'lib/raif/evals/eval_result.rb', line 21

def expectation_results
  @expectation_results
end

#judge_usage ⇒ Object (readonly)

Returns the value of attribute judge_usage.



21
22
23
 
# File 'lib/raif/evals/eval_result.rb', line 21

def judge_usage
  @judge_usage
end

#model_completions ⇒ Object (readonly)

Returns the value of attribute model_completions.



21
22
23
 
# File 'lib/raif/evals/eval_result.rb', line 21

def model_completions
  @model_completions
end

#overhead_usage ⇒ Object (readonly)

Returns the value of attribute overhead_usage.



21
22
23
 
# File 'lib/raif/evals/eval_result.rb', line 21

def overhead_usage
  @overhead_usage
end

#run_index ⇒ Object (readonly)

Returns the value of attribute run_index.



21
22
23
 
# File 'lib/raif/evals/eval_result.rb', line 21

def run_index
  @run_index
end

#scores ⇒ Object (readonly)

Returns the value of attribute scores.



21
22
23
 
# File 'lib/raif/evals/eval_result.rb', line 21

def scores
  @scores
end

#usage ⇒ Object (readonly)

Returns the value of attribute usage.



21
22
23
 
# File 'lib/raif/evals/eval_result.rb', line 21

def usage
  @usage
end

Instance Method Details

#add_expectation_result(result) ⇒ Object 



45
46
47
 
# File 'lib/raif/evals/eval_result.rb', line 45

def add_expectation_result(result)
  @expectation_results << result
end

#add_score(score_result) ⇒ Object

A score name is a metric the run summary aggregates by. Recording the same name twice for one eval would blend them into one row, hiding a regression in one behind an improvement in the other and narrowing the confidence interval on correlated values.



52
53
54
55
56
 
# File 'lib/raif/evals/eval_result.rb', line 52

def add_score(score_result)
  ensure_score_name_available!(score_result.name)

  @scores << score_result
end

#ensure_score_name_available!(name) ⇒ Object

Public so a caller about to spend money producing the value can ask first, as expect_llm_judge_score does: discovering the collision on the way back costs a request.

Raises:

  • (ArgumentError) 


60
61
62
63
64
65
 
# File 'lib/raif/evals/eval_result.rb', line 60

def ensure_score_name_available!(name)
  return unless @scores.any? { |score| score.name == name.to_s }

  raise ArgumentError, "score #{name.to_s.inspect} was already recorded for this eval. Give the two scores distinct " \
    "names (expect_llm_judge_score takes score_name:), or combine the values yourself and record one score."
end

#errored? ⇒ Boolean

An error is a missing measurement, not a bad one: a 429, a socket timeout, or a raise in setup says nothing about the quality of the model's output. Kept distinct from a failure all the way through aggregation, so a rate-limited afternoon does not read as a regression.

Returns:

  • (Boolean) 


99
100
101
 
# File 'lib/raif/evals/eval_result.rb', line 99

def errored?
  expectation_results.any?(&:error?)
end

#passed? ⇒ Boolean

Returns:

  • (Boolean) 


92
93
94
 
# File 'lib/raif/evals/eval_result.rb', line 92

def passed?
  expectation_results.all?(&:passed?)
end

#record_model_completions(completions, overhead: [], capture_mode: :full) ⇒ Object

Serialized into plain hashes here because the eval's transaction is about to be rolled back and the rows with it.

overhead is what setup and teardown spent. Kept out of #usage, which is the eval's own measurement, but summed so the run's cost total is not short of what the run cost. Usage only: the prompt and response of a call that built a fixture is not worth exporting.



73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
 
# File 'lib/raif/evals/eval_result.rb', line 73

def record_model_completions(completions, overhead: [], capture_mode: :full)
  serialized = Array(completions).map { |mc| serialize_model_completion(mc) }
  @usage = compute_usage(serialized)
  @overhead_usage = compute_usage(Array(overhead).map { |mc| serialize_model_completion(mc) })
  # A subset of #usage rather than a slice taken out of it: the eval did spend this, so the
  # run's cost total must keep counting it. The split is for the comparison.
  @judge_usage = compute_usage(serialized.select { |mc| mc[:judge] })
  @model_completions_captured = capture_mode.to_sym != :none

  @model_completions = case capture_mode.to_sym
  when :none
    []
  when :summary
    serialized.map { |mc| mc.except(*COMPLETION_TEXT_KEYS) }
  else
    serialized
  end
end

#to_h ⇒ Object 



103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
 
# File 'lib/raif/evals/eval_result.rb', line 103

def to_h
  {
    description: description,
    eval_id: eval_id,
    eval_index: eval_index,
    run_index: run_index,
    case_id: case_id,
    passed: passed?,
    # Omitted when false, for the reason overhead_usage is omitted when empty.
    errored: (true if errored?),
    expectation_results: expectation_results.map(&:to_h),
    scores: (scores.map(&:to_h) if scores.any?),
    usage: usage,
    # Omitted rather than zeroed: setup making no LLM calls is the normal case, and the key
    # would be noise in every result those runs wrote.
    overhead_usage: (overhead_usage if overhead_usage[:model_completions].positive?),
    # Omitted for the same reason, and for the same reason an absent key is not a zero: a
    # result written before the judge was tagged has unknown judge spend, not none.
    judge_usage: (judge_usage if judge_usage[:model_completions].positive?),
    model_completions: (model_completions if @model_completions_captured)
  }.compact
end