Class: Raif::ModelManifest::SmokeObservations

Inherits:
Object
  • Object
show all
Defined in:
lib/raif/model_manifest/smoke_observations.rb

Constant Summary collapse

SCHEMA_VERSION = 
1
RECORDABLE_POSITIVE_CAPABILITIES =

The complete set of capabilities bin/smoke --record may write as a durable pass; the single source of truth referenced by Smoke::Policy and Smoke::ObservationRecorder. temperature is intentionally absent: an accepted request does not prove the model honored the parameter.

%i[
  completion structured_outputs native_tool_use streaming
  streaming_tool_calls batch_inference images pdfs
  provider_managed_tools embedding
].freeze
DERIVED_CAPABILITIES =

Capabilities that are not literal manifest capability keys: their "claimed" state is derived from other capabilities. They are only ever offered as candidates when their preconditions are claimed, so their current claim is true by construction (for :embedding this is also the only option: EmbeddingEntry has no claimed_value method to call).

%i[completion streaming_tool_calls embedding].freeze

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(models) ⇒ SmokeObservations

Returns a new instance of SmokeObservations.



109
110
111
112
 
# File 'lib/raif/model_manifest/smoke_observations.rb', line 109

def initialize(models)
  @models = deep_freeze(models)
  freeze
end

Class Method Details

.load(dir: Raif::Engine.root.join("model_smoke_results")) ⇒ Object 



33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
 
# File 'lib/raif/model_manifest/smoke_observations.rb', line 33

def self.load(dir: Raif::Engine.root.join("model_smoke_results"))
  models = {}

  Dir[File.join(dir, "*.json")].sort.each do |path|
    data = parse_json(path)
    validate_schema_version!(data, path)

    data.fetch("models", {}).each do |model_key, capabilities|
      capabilities.each do |capability_name, record|
        checked_at = parse_checked_at(record["checked_at"], path: path, model_key: model_key, capability: capability_name)

        (models[model_key.to_sym] ||= {})[capability_name.to_sym] = {
          claimed: record.fetch("claimed"),
          result: record.fetch("result"),
          checked_at: checked_at
        }
      end
    end
  end

  new(models)
end

.normalize_claim(value) ⇒ Object

The single normalization both Smoke::ObservationRecorder (writing) and #fresh? (reading back, on both the stored and the live side) run a claim through, so a claim's shape never depends on which side of a JSON round trip it is on. A boolean claim passes through unchanged. An array claim (only provider_managed_tools today) is a symbol array on a live manifest entry but a string array once it has been through JSON, so both sides are stringified and sorted before comparison.



103
104
105
106
107
 
# File 'lib/raif/model_manifest/smoke_observations.rb', line 103

def self.normalize_claim(value)
  return value.map(&:to_s).sort if value.is_a?(Array)

  value
end

.recordable_candidates(entry) ⇒ Object

EmbeddingEntry -> [:embedding]; an LLM entry -> :completion plus every positively claimed recordable capability, plus :streaming_tool_calls only when streaming AND native_tool_use are both claimed, plus :provider_managed_tools only when the declared tool list is non-empty.



60
61
62
63
64
65
66
67
68
69
70
71
72
73
 
# File 'lib/raif/model_manifest/smoke_observations.rb', line 60

def self.recordable_candidates(entry)
  return [:embedding] if entry.is_a?(EmbeddingEntry)

  candidates = [:completion]
  candidates << :structured_outputs if entry.claimed_value("structured_outputs")
  candidates << :native_tool_use if entry.claimed_value("native_tool_use")
  candidates << :streaming if entry.claimed_value("streaming")
  candidates << :streaming_tool_calls if entry.claimed_value("streaming") && entry.claimed_value("native_tool_use")
  candidates << :batch_inference if entry.claimed_value("batch_inference")
  candidates << :images if entry.claimed_value("images")
  candidates << :pdfs if entry.claimed_value("pdfs")
  candidates << :provider_managed_tools if entry.claimed_value("provider_managed_tools")&.any?
  candidates
end

Instance Method Details

#fresh?(entry, capability, stale_after_days: 30, now: Time.now.utc) ⇒ Boolean

Returns:

  • (Boolean) 


114
115
116
117
118
119
120
121
122
 
# File 'lib/raif/model_manifest/smoke_observations.rb', line 114

def fresh?(entry, capability, stale_after_days: 30, now: Time.now.utc)
  capability = capability.to_sym
  record = @models.dig(entry.key, capability)
  return false unless record
  return false if record[:result] != "pass"
  return false if self.class.normalize_claim(record[:claimed]) != self.class.normalize_claim(current_claim(entry, capability))

  !stale_by_age?(record[:checked_at], stale_after_days: stale_after_days, now: now)
end

#stale_capabilities(entry, stale_after_days: 30, now: Time.now.utc) ⇒ Object 



124
125
126
 
# File 'lib/raif/model_manifest/smoke_observations.rb', line 124

def stale_capabilities(entry, stale_after_days: 30, now: Time.now.utc)
  self.class.recordable_candidates(entry).reject { |cap| fresh?(entry, cap, stale_after_days:, now:) }
end