Class: Raif::Evals::LlmJudge

Inherits:
Task
  • Object
show all
Defined in:
app/models/raif/evals/llm_judge.rb

Constant Summary

Constants included from Concerns::LlmResponseParsing

Concerns::LlmResponseParsing::ASCII_CONTROL_CHARS

Instance Attribute Summary

Attributes inherited from Task

#files, #images

Class Method Summary collapse

Instance Method Summary collapse

Methods inherited from Task

build_for_batch, #build_prompt, #build_system_prompt, find_sti_class, json_response_schema, #json_response_schema, #messages, #prepare_for_batch!, #process_completion!, prompt, #prompt_studio_task_attributes, #re_run, run, #run, #status, system_prompt

Methods included from Concerns::RunWith

deserialize_run_with_value, gid_string?, locate_gid, serialize_run_with_value

Methods included from Concerns::JsonSchemaDefinition

#schema_for_instance

Methods included from Concerns::LlmResponseParsing

#parse_html_response, #parse_json_response, #parsed_response

Methods included from Concerns::HasRuntimeDuration

#runtime_duration, #runtime_duration_seconds, #runtime_ended_at

Methods included from Concerns::HasAvailableModelTools

#available_model_tools_map

Methods included from Concerns::HasRequestedLanguage

#requested_language_name, #system_prompt_language_preference

Methods included from Concerns::HasLlm

#llm

Methods included from Concerns::HasPromptTemplates

#build_prompt, #build_system_prompt

Class Method Details

.resolved_llm_model_key ⇒ Object

The model that will grade, which is the model under test unless a judge is configured. A class method because Raif::Evals::Run reports it before any judge exists to ask, and the two must not be able to disagree about who is judging.



56
57
58
 
# File 'app/models/raif/evals/llm_judge.rb', line 56

def self.resolved_llm_model_key
  Raif.config.evals_default_llm_judge_model_key.presence || Raif.config.default_llm_model_key
end

Instance Method Details

#default_llm_model_key ⇒ Object 



60
61
62
 
# File 'app/models/raif/evals/llm_judge.rb', line 60

def default_llm_model_key
  self.class.resolved_llm_model_key
end

#judgment_confidence ⇒ Object 



68
69
70
 
# File 'app/models/raif/evals/llm_judge.rb', line 68

def judgment_confidence
  parsed_response["confidence"] if completed?
end

#judgment_reasoning ⇒ Object 



64
65
66
 
# File 'app/models/raif/evals/llm_judge.rb', line 64

def judgment_reasoning
  parsed_response["reasoning"] if completed?
end

#low_confidence? ⇒ Boolean

Returns:

  • (Boolean) 


72
73
74
 
# File 'app/models/raif/evals/llm_judge.rb', line 72

def low_confidence?
  judgment_confidence && judgment_confidence < 0.5
end