Class: Raif::Evals::EvalSetCoordinator

Inherits:
Object
  • Object
show all
Defined in:
lib/raif/evals/eval_set_coordinator.rb

Overview

Everything one eval set needs to run that is not the running of an eval: resolving its datasets, listing the executions it still owes, and dispatching them.

Not an EvalSet, because an EvalSet instance is single-use by construction: #run_eval writes the case and result onto it so #expect and #score can reach them. One of these is long-lived per eval set, and holds the collaborators shared across the whole run.

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(eval_set_class:, output: $stdout, run_log: nil, writer: nil, header: nil, cases: nil, sample: nil, seed: nil) ⇒ EvalSetCoordinator

Returns a new instance of EvalSetCoordinator.

Parameters:

  • eval_set_class (Class) —

    the Raif::Evals::EvalSet subclass being run.

  • writer (Raif::Evals::ConsoleWriter, nil) (defaults to: nil) —

    serializes this set's output with whatever else is writing to the same console. Defaults to an unbuffered writer of its own, which is what a host app calling EvalSet.run directly gets.

  • header (Array(Object, String), nil) (defaults to: nil) —

    a [key, line] pair the writer prints once, before this eval set's first line. Raif::Evals::Run uses it for the eval set banner.

  • cases (Array<String>, nil) (defaults to: nil) —

    restrict every dataset to these case ids (--cases).

  • sample (Integer, nil) (defaults to: nil) —

    draw this many cases from each dataset (--sample).

  • seed (Integer, nil) (defaults to: nil) —

    the seed that draw uses (--seed). These three scope the whole coordinator rather than any one call to it - see #selected_cases.



28
29
30
31
32
33
34
35
36
37
 
# File 'lib/raif/evals/eval_set_coordinator.rb', line 28

def initialize(eval_set_class:, output: $stdout, run_log: nil, writer: nil, header: nil, cases: nil, sample: nil, seed: nil)
  @eval_set_class = eval_set_class
  @output = output
  @run_log = run_log
  @writer = writer || ConsoleWriter.new(output)
  @header = header
  @cases = cases
  @sample = sample
  @seed = seed
end

Instance Attribute Details

#cases ⇒ Object (readonly)

Returns the value of attribute cases.



12
13
14
 
# File 'lib/raif/evals/eval_set_coordinator.rb', line 12

def cases
  @cases
end

#eval_set_class ⇒ Object (readonly)

Returns the value of attribute eval_set_class.



12
13
14
 
# File 'lib/raif/evals/eval_set_coordinator.rb', line 12

def eval_set_class
  @eval_set_class
end

#header ⇒ Object (readonly)

Returns the value of attribute header.



12
13
14
 
# File 'lib/raif/evals/eval_set_coordinator.rb', line 12

def header
  @header
end

#output ⇒ Object (readonly)

Returns the value of attribute output.



12
13
14
 
# File 'lib/raif/evals/eval_set_coordinator.rb', line 12

def output
  @output
end

#run_log ⇒ Object

Assignable because Raif::Evals::Run builds its coordinators before its run log: the log's header records dataset fingerprints only a resolved coordinator can produce.



16
17
18
 
# File 'lib/raif/evals/eval_set_coordinator.rb', line 16

def run_log
  @run_log
end

#sample ⇒ Object (readonly)

Returns the value of attribute sample.



12
13
14
 
# File 'lib/raif/evals/eval_set_coordinator.rb', line 12

def sample
  @sample
end

#seed ⇒ Object (readonly)

Returns the value of attribute seed.



12
13
14
 
# File 'lib/raif/evals/eval_set_coordinator.rb', line 12

def seed
  @seed
end

#writer ⇒ Object (readonly)

Returns the value of attribute writer.



12
13
14
 
# File 'lib/raif/evals/eval_set_coordinator.rb', line 12

def writer
  @writer
end

Instance Method Details

#dataset_fingerprints ⇒ Object

What this eval set's datasets held, for the run's configuration block. One entry per dataset, naming it, how many cases it has, and a digest of their contents - see Raif::Evals::Dataset#digest.



110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
 
# File 'lib/raif/evals/eval_set_coordinator.rb', line 110

def dataset_fingerprints
  datasets.map do |name, dataset|
    selected = selected_cases[name].count

    {
      eval_set: eval_set_class.name,
      name: name.to_s,
      cases: dataset.size,
      digest: dataset.digest,
      # Only when a selection narrowed the dataset, so a full run does not repeat `cases` in
      # every entry.
      selected: (selected unless selected == dataset.size)
    }.compact
  end
end

#executions_for(eval_definition, repeats: 1) ⇒ Object

The pending executions of one eval definition, in dataset order then repeat order.



76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
 
# File 'lib/raif/evals/eval_set_coordinator.rb', line 76

def executions_for(eval_definition, repeats: 1)
  eval_cases = selected_cases_for(eval_definition)
  eval_id = eval_definition.id
  run_indexes = repeat_run_indexes(repeats)

  if eval_cases.nil?
    return run_indexes.filter_map do |run_index|
      next if already_recorded?(eval_id, nil, run_index)

      Execution.new(eval_definition: eval_definition, run_index: run_index)
    end
  end

  # Widened over every selected case, not just the pending ones, so a resumed run's lines
  # stay aligned with the ones already printed.
  case_id_width = eval_cases.map { |eval_case| eval_case.id.length }.max

  eval_cases.flat_map do |eval_case|
    run_indexes.filter_map do |run_index|
      next if already_recorded?(eval_id, eval_case.id, run_index)

      Execution.new(
        eval_definition: eval_definition,
        eval_case: eval_case,
        run_index: run_index,
        case_id_width: case_id_width
      )
    end
  end
end

#pending_executions(repeats: 1) ⇒ Object

Every execution this eval set still owes, across all of its evals, without running any of them. Listing them rather than running them is what lets Raif::Evals::Run build one work list across every set.



51
52
53
54
55
 
# File 'lib/raif/evals/eval_set_coordinator.rb', line 51

def pending_executions(repeats: 1)
  eval_set_class.evals.flat_map do |eval_definition|
    executions_for(eval_definition, repeats: repeats)
  end
end

#planned_keys(eval_definitions: nil, repeats: 1) ⇒ Object

Every execution key this eval set covers, whether or not the run log already holds a result for it: the plan, where #executions_for is whatever is left of it. Deliberately blind to the log, since this is what the log is started with, before there is one to consult.

Parameters:

  • eval_definitions (Array<Raif::Evals::EvalDefinition>, nil) (defaults to: nil) —

    restrict the plan to these evals, which is what a file path given with a line number does.



63
64
65
66
67
68
69
70
71
72
73
 
# File 'lib/raif/evals/eval_set_coordinator.rb', line 63

def planned_keys(eval_definitions: nil, repeats: 1)
  (eval_definitions || eval_set_class.evals).flat_map do |eval_definition|
    eval_cases = selected_cases_for(eval_definition) || [nil]

    eval_cases.flat_map do |eval_case|
      repeat_run_indexes(repeats).map do |run_index|
        RunLog.key(eval_id: eval_definition.id, case_id: eval_case&.id, run_index: run_index)
      end
    end
  end
end

#record_from_worker(execution, value) ⇒ Object

The other half, in the parent, with what #run_in_worker returned.



164
165
166
167
168
169
 
# File 'lib/raif/evals/eval_set_coordinator.rb', line 164

def record_from_worker(execution, value)
  writer.capture(headers: headers_for(execution)) do |execution_output|
    execution_output.write(value[:output])
    run_log&.record(eval_set: eval_set_class.name, result: value[:result])
  end
end

#result_order ⇒ Object

Where each [eval_index, case_id] pair sits in this set's definition order, for putting results that completed in another order back into it. Keyed on every selected case, so a resumed run orders the results it inherited too. A position rather than the case id, since sorting on the id would replace the dataset author's order with alphabetical order.



130
131
132
133
134
135
 
# File 'lib/raif/evals/eval_set_coordinator.rb', line 130

def result_order
  eval_set_class.evals.flat_map do |eval_definition|
    eval_cases = selected_cases_for(eval_definition) || [nil]
    eval_cases.map.with_index { |eval_case, position| [[eval_definition.index, eval_case&.id], position] }
  end.to_h
end

#run(repeats: 1) ⇒ Object

Runs everything this set still owes, in definition order. The path a host app calling EvalSet.run takes; Raif::Evals::Run instead collects #pending_executions across every set and dispatches them itself.



42
43
44
45
46
 
# File 'lib/raif/evals/eval_set_coordinator.rb', line 42

def run(repeats: 1)
  pending_executions(repeats: repeats).map do |execution|
    run_and_record(execution)
  end
end

#run_and_record(execution) ⇒ Object

Runs one execution and records its result, in this process. The serial path, where the execution's lines stream to the console as they are written.



139
140
141
142
143
144
145
146
147
148
149
150
151
 
# File 'lib/raif/evals/eval_set_coordinator.rb', line 139

def run_and_record(execution)
  eval_result = nil

  writer.capture(headers: headers_for(execution)) do |execution_output|
    eval_result = run_execution(execution, execution_output)

    # Recorded the moment it completes, so the run's spend survives an interrupt that
    # never reaches the results file.
    run_log&.record(eval_set: eval_set_class.name, result: eval_result)
  end

  eval_result
end

#run_in_worker(execution) ⇒ Object

The half of #run_and_record a forked worker does: runs the execution and returns what the parent needs to record it, as plain data that survives Marshal. The run log and the console belong to the parent, so nothing here writes to either.



156
157
158
159
160
161
 
# File 'lib/raif/evals/eval_set_coordinator.rb', line 156

def run_in_worker(execution)
  execution_output = StringIO.new
  eval_result = run_execution(execution, execution_output)

  { output: execution_output.string, result: RunLog.json_round_trip(eval_result.to_h) }
end