Class: Raif::ArchiveTasksJob

Inherits:
ArchiveJob show all
Defined in:
app/jobs/raif/archive_tasks_job.rb

Overview

Archives Raif::Task rows older than Raif.config.task_retention_period to Raif.config.archive_storage as gzip JSONL, then deletes them. Raif::ArchiveJob owns the batching, partitioning and cull machinery and documents the invariants; this class contributes the eligibility rules and the handling of the rows that point at a task.

A task's spend lives on its Raif::ModelCompletion's Raif::InferenceCostEvent, so this job has no durability guard of its own. What it has instead is the completion gate: it never culls a task whose completion row is still present.

Constant Summary collapse

KEY_PREFIX =

Key prefix when partitioning is unset.

"raif-archives/tasks"
RESOURCE_KEY_SEGMENT =

Resource segment below a partition prefix: raif-archives/partitions//tasks/

"tasks"
INVOCATIONS_KEY_PREFIX =

Model tool invocations travel with their task, in their own object under the task's partition prefix.

"raif-archives/model-tool-invocations"
INVOCATIONS_RESOURCE_KEY_SEGMENT = 
"model-tool-invocations"

Constants inherited from ArchiveJob

Raif::ArchiveJob::BATCH_RECORD_LIMIT, Raif::ArchiveJob::BATCH_UNCOMPRESSED_BYTE_LIMIT, Raif::ArchiveJob::PARTITIONS_PER_PASS, Raif::ArchiveJob::QUIESCENCE_PERIOD

Class Method Summary collapse

Methods inherited from ArchiveJob

base_scope, dry_run, partition_column, #perform, quiescent_scope, resource_table, terminal_scope, ungrouped_fallback?, validate_partition_column!

Class Method Details

.archived_class ⇒ Object 



26
27
28
 
# File 'app/jobs/raif/archive_tasks_job.rb', line 26

def archived_class
  Raif::Task
end

.dry_run_exclusions(cutoff) ⇒ Object 



73
74
75
76
77
78
 
# File 'app/jobs/raif/archive_tasks_job.rb', line 73

def dry_run_exclusions(cutoff)
  {
    excluded_by_live_model_completion: base_scope(cutoff).where(id: tasks_with_live_model_completion).count,
    excluded_by_prompt_studio: base_scope(cutoff).where(prompt_studio_referenced_sql).count
  }
end

.eligible_scope(cutoff) ⇒ Object

A task is safe to archive and delete only when ALL hold:

  • created_at is before the (job-frozen) retention cutoff
  • it has been quiescent: not updated within QUIESCENCE_PERIOD
  • completion gate: no Raif::ModelCompletion row still names it as its source. Raif::Task declares has_one :raif_model_completion with dependent: :destroy, which a bulk delete does not run, so culling a task out from under a live completion would leave that completion pointing at nothing - and the raif admin resolves a completion's source on every render. Waiting is free: the completion job archives the completion first, and this job takes the task on a later run.
  • prompt studio gate: no raif_prompt_studio_batch_run_items row references it. That table's source_task_id is NOT NULL and all three of its task foreign keys RESTRICT, so deleting a referenced task raises and takes the whole cull down with it. Prompt studio rows are never deleted, so a task a batch run touched is retained indefinitely, by design.

Nonterminal tasks (the residue of killed processes and crashed jobs) are archived like any other, so "every deleted task exists in an archive" holds without exception.



67
68
69
70
71
 
# File 'app/jobs/raif/archive_tasks_job.rb', line 67

def eligible_scope(cutoff)
  quiescent_scope(cutoff)
    .where.not(id: tasks_with_live_model_completion)
    .where.not(prompt_studio_referenced_sql)
end

.key_prefix ⇒ Object 



38
39
40
 
# File 'app/jobs/raif/archive_tasks_job.rb', line 38

def key_prefix
  KEY_PREFIX
end

.resource_key_segment ⇒ Object 



42
43
44
 
# File 'app/jobs/raif/archive_tasks_job.rb', line 42

def resource_key_segment
  RESOURCE_KEY_SEGMENT
end

.retention_config_name ⇒ Object 



34
35
36
 
# File 'app/jobs/raif/archive_tasks_job.rb', line 34

def retention_config_name
  "task_retention_period"
end

.retention_period ⇒ Object 



30
31
32
 
# File 'app/jobs/raif/archive_tasks_job.rb', line 30

def retention_period
  Raif.config.task_retention_period
end