Class: Raif::ArchiveModelCompletionsJob

Inherits:
ArchiveJob show all
Defined in:
app/jobs/raif/archive_model_completions_job.rb

Overview

Archives Raif::ModelCompletion rows older than Raif.config.model_completion_retention_period to Raif.config.archive_storage as gzip JSONL (see Raif::ArchiveSerializer), then deletes them, one Raif::Archive batch at a time. Raif::ArchiveJob owns the batching, partitioning and cull machinery and documents the invariants; this class contributes the eligibility rules.

Deliberately NOT handled: raif_model_completion_batches rows remain; they carry their own aggregated cost columns and nothing recomputes them from children after finalization.

Constant Summary collapse

KEY_PREFIX =

Key prefix when partitioning is unset.

"raif-archives/model-completions"
RESOURCE_KEY_SEGMENT =

Resource segment below a partition prefix: raif-archives/partitions//model-completions/

"model-completions"

Constants inherited from ArchiveJob

Raif::ArchiveJob::BATCH_RECORD_LIMIT, Raif::ArchiveJob::BATCH_UNCOMPRESSED_BYTE_LIMIT, Raif::ArchiveJob::PARTITIONS_PER_PASS, Raif::ArchiveJob::QUIESCENCE_PERIOD

Class Method Summary collapse

Methods inherited from ArchiveJob

base_scope, dry_run, partition_column, #perform, quiescent_scope, resource_table, terminal_scope, ungrouped_fallback?, validate_partition_column!

Class Method Details

.archived_class ⇒ Object 



22
23
24
 
# File 'app/jobs/raif/archive_model_completions_job.rb', line 22

def archived_class
  Raif::ModelCompletion
end

.dry_run_exclusions(cutoff) ⇒ Object 



77
78
79
80
81
82
83
84
85
86
 
# File 'app/jobs/raif/archive_model_completions_job.rb', line 77

def dry_run_exclusions(cutoff)
  {
    excluded_by_active_batch: base_scope(cutoff).where(id: active_batch_members).count,
    excluded_missing_cost_event: terminal_scope(cutoff).where.not(id: completions_with_cost_event).count,
    excluded_stale_cost_event: terminal_scope(cutoff)
      .where(id: completions_with_cost_event)
      .where.not(id: completions_with_fresh_cost_event).count,
    excluded_uncopied_citations: base_scope(cutoff).where(id: completions_with_uncopied_citations).count
  }
end

.eligible_scope(cutoff) ⇒ Object

A completion is safe to archive and delete only when ALL hold:

  • created_at is before the (job-frozen) retention cutoff
  • it has been quiescent: not updated within QUIESCENCE_PERIOD (legitimately active months-old completions can't exist since batch lifetime is capped, but the guard is cheap insurance)
  • it is not a member of a model completion batch that is still non-terminal (belt-and-suspenders alongside quiescence)
  • durability guard, TERMINAL rows only: its Raif::InferenceCostEvent exists AND is at least as fresh as the completion (event.updated_at >= completion.updated_at). A post-terminal update whose event re-sync failed leaves a stale event that missing-only repair would never revisit, so the repair job also re-syncs stale events; until then the row just waits.
  • nonterminal rows skip the durability guard: they never reached a terminal state, so no cost event exists and there is no spend to protect. These are orphaned pending rows from killed processes and crashed jobs (a third of one host's table in practice) that would otherwise be immortal. They are archived through the same path as everything else - NOT deleted outright, despite the temptation (no response, near-zero historical value): "every deleted completion exists in an archive" must hold without exception, and a delete-without-archive shortcut would be a second deletion semantics that weakens the invariant this job's safety rests on, to save pennies of mostly-redundant prompt storage.
  • durable-citations guard: its citations, if any, have been copied to its Raif::ConversationEntry source (protects hosts that haven't run the conversation entry backfill)


70
71
72
73
74
75
 
# File 'app/jobs/raif/archive_model_completions_job.rb', line 70

def eligible_scope(cutoff)
  quiescent_scope(cutoff)
    .where.not(id: active_batch_members)
    .where.not(id: terminal_without_fresh_cost_event(cutoff))
    .where.not(id: completions_with_uncopied_citations)
end

.key_prefix ⇒ Object 



34
35
36
 
# File 'app/jobs/raif/archive_model_completions_job.rb', line 34

def key_prefix
  KEY_PREFIX
end

.resource_key_segment ⇒ Object 



38
39
40
 
# File 'app/jobs/raif/archive_model_completions_job.rb', line 38

def resource_key_segment
  RESOURCE_KEY_SEGMENT
end

.retention_config_name ⇒ Object 



30
31
32
 
# File 'app/jobs/raif/archive_model_completions_job.rb', line 30

def retention_config_name
  "model_completion_retention_period"
end

.retention_period ⇒ Object 



26
27
28
 
# File 'app/jobs/raif/archive_model_completions_job.rb', line 26

def retention_period
  Raif.config.model_completion_retention_period
end