Class: Raif::ArchiveJob

Inherits:
ApplicationJob show all
Defined in:
app/jobs/raif/archive_job.rb

Overview

Abstract base for the archive-and-cull jobs. Subclasses (see Raif::ArchiveModelCompletionsJob, Raif::ArchiveTasksJob) name the resource and its eligibility rules; everything below - batching, partition fairness, upload, the tainted-partition abort and the cull transaction - is shared, so the safety invariants have exactly one implementation. Enabling, scheduling, storage requirements and partitioning are documented for operators at https://docs.raif.ai/learn_more/archiving.

The invariants this job exists to hold:

  • At-least-once archiving + never-delete-unarchived. A batch is deleted only after this run uploaded it, and a re-run after any crash writes a new object under a new key rather than resuming or overwriting. There is no persisted in-flight state to repair, ever.
  • The accepted cost of that is duplicate objects: a crash between the upload and the Raif::Archive insert leaves an object that no row references (those rows were not deleted, so they re-archive next run). Storage policy must therefore apply to whole prefixes and never be derived from raif_archives rows, which is also what makes Raif::Archive.purge_partition! complete.
  • Inert unless the host opts in: archive_enabled, an archive_storage adapter and a retention period must all be set, or perform returns without touching anything.
  • With partitioning, every object holds records from exactly one partition (see Raif::ArchivePartition). Per-partition erasure rests on that, so a record whose partition changes mid-upload taints the object and aborts the cull.

Subclasses implement the "Subclass contract" class methods below and the stamp_cost_events! hook; the dependent-resource hooks are optional.

Direct Known Subclasses

ArchiveModelCompletionsJob, ArchiveTasksJob

Constant Summary collapse

BATCH_RECORD_LIMIT =

Caps are constants, not config: public config is API; add a knob only when a host demonstrably needs it. Read through self, so a subclass can override one. A batch closes when EITHER cap is hit; the byte cap keeps wildly varying payloads (agent message arrays can be enormous) from building a multi-GB tempfile.

25_000
BATCH_UNCOMPRESSED_BYTE_LIMIT = 
512.megabytes
QUIESCENCE_PERIOD =

Rows updated more recently than this are ineligible for archiving, regardless of age.

24.hours
PARTITIONS_PER_PASS =

Distinct partitions fetched per round-robin pass. The listing query is a GROUP BY over the full multi-guard eligibility scope, the one potentially expensive query in the fairness design, so it is capped.

100

Class Method Summary collapse