Class: Raif::Evals::EvalDefinition

Inherits:
Object
  • Object
show all
Defined in:
lib/raif/evals/eval_definition.rb

Overview

One eval "..." do ... end declaration: the block, what it is called, and the dataset it runs over. The definition, not the outcome - running one of these once against one case produces an EvalResult.

index is the definition's position in its eval set, assigned when it registers. Results are grouped by it when repeats are collapsed into a pass rate, and sorted by it back into definition order. It is deliberately not the identity: see #id.

Constant Summary collapse

DESCRIPTION_SLUG_LIMIT =

Caps the readable half so a sentence-long description does not produce an id that wraps a terminal. Truncating it is safe: the digest, not the slug, is what makes an id unique.

60
DIGEST_LENGTH =

48 bits. A collision only matters within one eval set, and the duplicate-description check already refuses the realistic way that happens.

12

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(description:, block:, index:, eval_set_class:, declared_id: nil, dataset: nil, file: nil, line_number: nil) ⇒ EvalDefinition

Returns a new instance of EvalDefinition.



25
26
27
28
29
30
31
32
33
34
 
# File 'lib/raif/evals/eval_definition.rb', line 25

def initialize(description:, block:, index:, eval_set_class:, declared_id: nil, dataset: nil, file: nil, line_number: nil)
  @description = description
  @block = block
  @dataset = dataset
  @index = index
  @eval_set_class = eval_set_class
  @declared_id = declared_id
  @file = file
  @line_number = line_number
end

Instance Attribute Details

#block ⇒ Object (readonly)

Returns the value of attribute block.



23
24
25
 
# File 'lib/raif/evals/eval_definition.rb', line 23

def block
  @block
end

#dataset ⇒ Object (readonly)

Returns the value of attribute dataset.



23
24
25
 
# File 'lib/raif/evals/eval_definition.rb', line 23

def dataset
  @dataset
end

#declared_id ⇒ Object (readonly)

Returns the value of attribute declared_id.



23
24
25
 
# File 'lib/raif/evals/eval_definition.rb', line 23

def declared_id
  @declared_id
end

#description ⇒ Object (readonly)

Returns the value of attribute description.



23
24
25
 
# File 'lib/raif/evals/eval_definition.rb', line 23

def description
  @description
end

#eval_set_class ⇒ Object (readonly)

Returns the value of attribute eval_set_class.



23
24
25
 
# File 'lib/raif/evals/eval_definition.rb', line 23

def eval_set_class
  @eval_set_class
end

#file ⇒ Object (readonly)

Returns the value of attribute file.



23
24
25
 
# File 'lib/raif/evals/eval_definition.rb', line 23

def file
  @file
end

#index ⇒ Object (readonly)

Returns the value of attribute index.



23
24
25
 
# File 'lib/raif/evals/eval_definition.rb', line 23

def index
  @index
end

#line_number ⇒ Object (readonly)

Returns the value of attribute line_number.



23
24
25
 
# File 'lib/raif/evals/eval_definition.rb', line 23

def line_number
  @line_number
end

Instance Method Details

#dataset? ⇒ Boolean

Returns:

  • (Boolean) 


53
54
55
 
# File 'lib/raif/evals/eval_definition.rb', line 53

def dataset?
  !dataset.nil?
end

#id ⇒ Object

What identifies this eval across runs: the key evals:compare joins on and --resume skips already-recorded work by. Position cannot do that job, since inserting an eval block shifts every index below it and a comparison would then join one eval's baseline to another eval's candidate with nothing looking wrong.

Derived rather than declared: class name, a slug of the description, and a digest over both. The digest is the identifying half and is taken over the description verbatim, so "handles > 100 items" and "handles < 100 items" stay distinct despite slugging identically. Rewording a description therefore produces a new id - id: is the escape hatch when the wording changed but the eval did not.

Computed on first use rather than at registration, because a class assigned to a constant after its body runs - or stub_const - has no name while its evals are registering.



49
50
51
 
# File 'lib/raif/evals/eval_definition.rb', line 49

def id
  @id ||= "#{eval_set_name}##{declared_id || derived_suffix}"
end