Class: Raif::Evals::Dataset

Inherits:
Object
  • Object
show all
Defined in:
lib/raif/evals/dataset.rb

Overview

An ordered, named collection of EvalCases, built from whatever the eval set's dataset block returned.

Validation raises rather than warns, and runs before any eval executes: a dataset whose cases cannot be told apart produces results that cannot be joined against a previous run, and finding that out after paying for the inference is too late.

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(name:, cases:) ⇒ Dataset

Returns a new instance of Dataset.



18
19
20
21
22
 
# File 'lib/raif/evals/dataset.rb', line 18

def initialize(name:, cases:)
  @name = name
  @cases = build_cases(cases)
  validate_unique_ids!
end

Instance Attribute Details

#cases ⇒ Object (readonly)

Returns the value of attribute cases.



16
17
18
 
# File 'lib/raif/evals/dataset.rb', line 16

def cases
  @cases
end

#name ⇒ Object (readonly)

Returns the value of attribute name.



16
17
18
 
# File 'lib/raif/evals/dataset.rb', line 16

def name
  @name
end

Instance Method Details

#digest ⇒ Object

A fingerprint of what this dataset holds, recorded in the run's configuration so two runs can tell whether they measured the same inputs. Without it an edited case reads as a model regression: evals:compare joins on case id, so the same id carrying a different input is reported as the model behaving differently on the same one.

Over the whole dataset rather than the selected cases, since --cases, --sample and --seed are recorded beside it and already say which of these ran.



35
36
37
 
# File 'lib/raif/evals/dataset.rb', line 35

def digest
  @digest ||= "sha256:#{Digest::SHA256.hexdigest(canonical_cases)}"
end

#only(ids) ⇒ Object 



55
56
57
 
# File 'lib/raif/evals/dataset.rb', line 55

def only(ids)
  select_cases(ids: ids)
end

#sample(count, seed: nil) ⇒ Object 



59
60
61
 
# File 'lib/raif/evals/dataset.rb', line 59

def sample(count, seed: nil)
  select_cases(sample: count, seed: seed)
end

#select_cases(ids: nil, sample: nil, seed: nil) ⇒ Object

ids and sample are run-wide (--cases / --sample), so an id belonging to another eval set's dataset filters this one to nothing rather than raising. Run#execute errors when a selection matched no case anywhere.



42
43
44
45
46
47
48
49
50
51
52
53
 
# File 'lib/raif/evals/dataset.rb', line 42

def select_cases(ids: nil, sample: nil, seed: nil)
  selected = cases

  if ids
    wanted = Array(ids).map(&:to_s).to_set
    selected = selected.select { |eval_case| wanted.include?(eval_case.id) }
  end

  selected = sample_cases(selected, sample.to_i, seed) if sample && sample.to_i < selected.length

  selected
end

#size ⇒ Object 



24
25
26
 
# File 'lib/raif/evals/dataset.rb', line 24

def size
  cases.length
end