Class: Raif::Evals::EvalSetCoordinator

Inherits:
Object
  • Object
show all
Defined in:
lib/raif/evals/eval_set_coordinator.rb

Overview

Everything one eval set needs to run that is not the running of an eval: resolving its datasets, listing the executions it still owes, and dispatching them.

Not an EvalSet, because an EvalSet instance is single-use by construction: #run_eval writes the case and result onto it so #expect and #score can reach them. One of these is long-lived per eval set, and holds the collaborators shared across the whole run.

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(eval_set_class:, output: $stdout, run_log: nil, writer: nil, header: nil, cases: nil, sample: nil, seed: nil) ⇒ EvalSetCoordinator

Returns a new instance of EvalSetCoordinator.

Parameters:

  • eval_set_class (Class) —

    the Raif::Evals::EvalSet subclass being run.

  • writer (Raif::Evals::ConsoleWriter, nil) (defaults to: nil) —

    serializes this set's output with whatever else is writing to the same console. Defaults to an unbuffered writer of its own, which is what a host app calling EvalSet.run directly gets.

  • header (Array(Object, String), nil) (defaults to: nil) —

    a [key, line] pair the writer prints once, before this eval set's first line. Raif::Evals::Run uses it for the eval set banner.

  • cases (Array<String>, nil) (defaults to: nil) —

    restrict every dataset to these case ids (--cases).

  • sample (Integer, nil) (defaults to: nil) —

    draw this many cases from each dataset (--sample).

  • seed (Integer, nil) (defaults to: nil) —

    the seed that draw uses (--seed). These three scope the whole coordinator rather than any one call to it - see #selected_cases.



28
29
30
31
32
33
34
35
36
37
 
# File 'lib/raif/evals/eval_set_coordinator.rb', line 28

def initialize(eval_set_class:, output: $stdout, run_log: nil, writer: nil, header: nil, cases: nil, sample: nil, seed: nil)
  @eval_set_class = eval_set_class
  @output = output
  @run_log = run_log
  @writer = writer || ConsoleWriter.new(output)
  @header = header
  @cases = cases
  @sample = sample
  @seed = seed
end

Instance Attribute Details

#cases ⇒ Object (readonly)

Returns the value of attribute cases.



12
13
14
 
# File 'lib/raif/evals/eval_set_coordinator.rb', line 12

def cases
  @cases
end

#eval_set_class ⇒ Object (readonly)

Returns the value of attribute eval_set_class.



12
13
14
 
# File 'lib/raif/evals/eval_set_coordinator.rb', line 12

def eval_set_class
  @eval_set_class
end

#header ⇒ Object (readonly)

Returns the value of attribute header.



12
13
14
 
# File 'lib/raif/evals/eval_set_coordinator.rb', line 12

def header
  @header
end

#output ⇒ Object (readonly)

Returns the value of attribute output.



12
13
14
 
# File 'lib/raif/evals/eval_set_coordinator.rb', line 12

def output
  @output
end

#run_log ⇒ Object

Assignable because Raif::Evals::Run builds its coordinators before its run log: the log's header records dataset fingerprints only a resolved coordinator can produce.



16
17
18
 
# File 'lib/raif/evals/eval_set_coordinator.rb', line 16

def run_log
  @run_log
end

#sample ⇒ Object (readonly)

Returns the value of attribute sample.



12
13
14
 
# File 'lib/raif/evals/eval_set_coordinator.rb', line 12

def sample
  @sample
end

#seed ⇒ Object (readonly)

Returns the value of attribute seed.



12
13
14
 
# File 'lib/raif/evals/eval_set_coordinator.rb', line 12

def seed
  @seed
end

#writer ⇒ Object (readonly)

Returns the value of attribute writer.



12
13
14
 
# File 'lib/raif/evals/eval_set_coordinator.rb', line 12

def writer
  @writer
end

Instance Method Details

#dataset_fingerprints ⇒ Object

What this eval set's datasets held, for the run's configuration block. One entry per dataset, naming it, how many cases it has, and a digest of their contents - see Raif::Evals::Dataset#digest.



110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
 
# File 'lib/raif/evals/eval_set_coordinator.rb', line 110

def dataset_fingerprints
  datasets.map do |name, dataset|
    selected = selected_cases[name].count

    {
      eval_set: eval_set_class.name,
      name: name.to_s,
      cases: dataset.size,
      digest: dataset.digest,
      # Only when a selection narrowed the dataset, so a full run does not repeat `cases` in
      # every entry.
      selected: (selected unless selected == dataset.size)
    }.compact
  end
end

#executions_for(eval_definition, repeats: 1) ⇒ Object

The pending executions of one eval definition, in dataset order then repeat order.



76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
 
# File 'lib/raif/evals/eval_set_coordinator.rb', line 76

def executions_for(eval_definition, repeats: 1)
  eval_cases = selected_cases_for(eval_definition)
  eval_id = eval_definition.id
  run_indexes = repeat_run_indexes(repeats)

  if eval_cases.nil?
    return run_indexes.filter_map do |run_index|
      next if already_recorded?(eval_id, nil, run_index)

      Execution.new(eval_definition: eval_definition, run_index: run_index)
    end
  end

  # Widened over every selected case, not just the pending ones, so a resumed run's lines
  # stay aligned with the ones already printed.
  case_id_width = eval_cases.map { |eval_case| eval_case.id.length }.max

  eval_cases.flat_map do |eval_case|
    run_indexes.filter_map do |run_index|
      next if already_recorded?(eval_id, eval_case.id, run_index)

      Execution.new(
        eval_definition: eval_definition,
        eval_case: eval_case,
        run_index: run_index,
        case_id_width: case_id_width
      )
    end
  end
end

#pending_executions(repeats: 1) ⇒ Object

Every execution this eval set still owes, across all of its evals, without running any of them. Listing them rather than running them is what lets Raif::Evals::Run build one work list across every set.



51
52
53
54
55
 
# File 'lib/raif/evals/eval_set_coordinator.rb', line 51

def pending_executions(repeats: 1)
  eval_set_class.evals.flat_map do |eval_definition|
    executions_for(eval_definition, repeats: repeats)
  end
end

#planned_keys(eval_definitions: nil, repeats: 1) ⇒ Object

Every execution key this eval set covers, whether or not the run log already holds a result for it: the plan, where #executions_for is whatever is left of it. Deliberately blind to the log, since this is what the log is started with, before there is one to consult.

Parameters:

  • eval_definitions (Array<Raif::Evals::EvalDefinition>, nil) (defaults to: nil) —

    restrict the plan to these evals, which is what a file path given with a line number does.



63
64
65
66
67
68
69
70
71
72
73
 
# File 'lib/raif/evals/eval_set_coordinator.rb', line 63

def planned_keys(eval_definitions: nil, repeats: 1)
  (eval_definitions || eval_set_class.evals).flat_map do |eval_definition|
    eval_cases = selected_cases_for(eval_definition) || [nil]

    eval_cases.flat_map do |eval_case|
      repeat_run_indexes(repeats).map do |run_index|
        RunLog.key(eval_id: eval_definition.id, case_id: eval_case&.id, run_index: run_index)
      end
    end
  end
end

#result_order ⇒ Object

Where each [eval_index, case_id] pair sits in this set's definition order, for putting results that completed in another order back into it. Keyed on every selected case, so a resumed run orders the results it inherited too. A position rather than the case id, since sorting on the id would replace the dataset author's order with alphabetical order.



130
131
132
133
134
135
 
# File 'lib/raif/evals/eval_set_coordinator.rb', line 130

def result_order
  eval_set_class.evals.flat_map do |eval_definition|
    eval_cases = selected_cases_for(eval_definition) || [nil]
    eval_cases.map.with_index { |eval_case, position| [[eval_definition.index, eval_case&.id], position] }
  end.to_h
end

#run(repeats: 1) ⇒ Object

Runs everything this set still owes, in definition order. The path a host app calling EvalSet.run takes; Raif::Evals::Run instead collects #pending_executions across every set and dispatches them itself.



42
43
44
45
46
 
# File 'lib/raif/evals/eval_set_coordinator.rb', line 42

def run(repeats: 1)
  pending_executions(repeats: repeats).map do |execution|
    run_and_record(execution)
  end
end

#run_and_record(execution) ⇒ Object

Runs one execution and records its result. The unit of work Raif::Evals::Run hands to a worker thread, so everything it touches has to be safe to call concurrently: the run log takes a lock, and console output goes through a writer that flushes as one block.



140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
 
# File 'lib/raif/evals/eval_set_coordinator.rb', line 140

def run_and_record(execution)
  eval_result = nil

  writer.capture(headers: headers_for(execution)) do |execution_output|
    # A fresh eval set per execution: run_eval writes the current case and result onto the
    # instance, so a shared one would allow only one execution in flight.
    eval_result = eval_set_class.new(output: execution_output).run_eval(
      execution.eval_definition,
      eval_case: execution.eval_case,
      run_index: execution.run_index,
      case_id_width: execution.case_id_width
    )

    # Recorded the moment it completes, so the run's spend survives an interrupt that
    # never reaches the results file.
    run_log&.record(eval_set: eval_set_class.name, result: eval_result)
  end

  eval_result
end