Class: Raif::Evals::RunPlan

Inherits:
Object
  • Object
show all
Defined in:
lib/raif/evals/run_plan.rb

Overview

The executions a run set out to perform, fixed once its datasets are resolved and before its first eval runs.

A run log records what a run has already done. On its own that says nothing about what the run still owes, so anything narrower than the original invocation - --resume given one eval set file - would drain its own shorter work list, write the results file and delete the log while the rest of the run was still outstanding. The plan is what lets the log answer whether the run is finished, rather than the invocation answering it for itself.

A key is Raif::Evals::RunLog.key: which eval block, against which case, on which repeat.

Constant Summary collapse

VERSION =

The shape of a persisted plan. A log written under a version this code cannot read is not resumable, since these keys are how a resume tells what is done from what is outstanding.

1

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(keys:) ⇒ RunPlan

Returns a new instance of RunPlan.



24
25
26
 
# File 'lib/raif/evals/run_plan.rb', line 24

def initialize(keys:)
  @keys = keys.map { |key| self.class.normalize(key) }.uniq
end

Instance Attribute Details

#keys ⇒ Object (readonly)

Returns the value of attribute keys.



22
23
24
 
# File 'lib/raif/evals/run_plan.rb', line 22

def keys
  @keys
end

Class Method Details

.from_h(data) ⇒ Object

nil for anything this version cannot read, including a log written before plans existed. Raif::Evals::RunLog is what reports on that, so that every reason a resume is refused is phrased in one place.



31
32
33
34
35
36
 
# File 'lib/raif/evals/run_plan.rb', line 31

def self.from_h(data)
  return unless data.is_a?(Hash)
  return unless data[:version].to_i == VERSION

  new(keys: Array(data[:keys]))
end

.normalize(key) ⇒ Object

JSON has no symbols and no tuples, so a key read back out of a log is an array of strings, integers and nulls. Coerced on the way in rather than compared loosely at every use.



40
41
42
43
44
 
# File 'lib/raif/evals/run_plan.rb', line 40

def self.normalize(key)
  eval_id, case_id, run_index = key

  [eval_id.to_s, case_id&.to_s, run_index&.to_i]
end

Instance Method Details

#additions(other) ⇒ Object

The keys of another plan this one does not already hold, in that plan's order. A resumed invocation can turn up executions the original run never planned - an eval block added to a file while the run was interrupted - and those are work the run now owes too.



68
69
70
 
# File 'lib/raif/evals/run_plan.rb', line 68

def additions(other)
  other.keys.reject { |key| include?(key) }
end

#include?(key) ⇒ Boolean

Returns:

  • (Boolean) 


54
55
56
 
# File 'lib/raif/evals/run_plan.rb', line 54

def include?(key)
  key_set.include?(self.class.normalize(key))
end

#outstanding(recorded_keys) ⇒ Object

The planned keys no result has been recorded for, in plan order.



59
60
61
62
63
 
# File 'lib/raif/evals/run_plan.rb', line 59

def outstanding(recorded_keys)
  recorded = recorded_keys.map { |key| self.class.normalize(key) }.to_set

  keys.reject { |key| recorded.include?(key) }
end

#plus(additional_keys) ⇒ Object 



72
73
74
 
# File 'lib/raif/evals/run_plan.rb', line 72

def plus(additional_keys)
  self.class.new(keys: keys + additional_keys)
end

#size ⇒ Object 



50
51
52
 
# File 'lib/raif/evals/run_plan.rb', line 50

def size
  keys.length
end

#to_h ⇒ Object 



46
47
48
 
# File 'lib/raif/evals/run_plan.rb', line 46

def to_h
  { version: VERSION, keys: keys }
end