Class: Raif::ArchiveSerializer

Inherits:
Object
  • Object
show all
Defined in:
app/models/raif/archive_serializer.rb

Overview

Serializes an ordered relation of records to a gzip-compressed JSON Lines tempfile for archival:

  • Line 1 is a versioned manifest: resource type, table, columns, cutoff, the serialized id range/count, and generation time.
  • Lines 2..n are one record per line: the full record.attributes as JSON, complete raw attributes including ids and polymorphic references, NOT a reduced projection. Round-trippable for manual recovery or re-insertion with original ids.

Serialization stops early when the running uncompressed byte total reaches byte_limit (completion payloads vary wildly; a pure record count could otherwise build a multi-GB file). At least one record is always written so an oversized single record still archives. The manifest must be line 1 but its record count/range aren't known until the byte cap resolves, so records spool uncompressed to a scratch tempfile first, then manifest + records are gzip-written to the final file. record_ids in the result is the ids ACTUALLY written - the caller must upload, record, and delete exactly that subset; anything cut off by the cap simply stays eligible for a later batch.

Resource-agnostic by design: nothing here may know about Raif::ModelCompletion specifics, because Raif::Task archiving (and any host resource) reuses this class.

Constant Summary collapse

MANIFEST_VERSION = 
1

Instance Method Summary collapse

Constructor Details

#initialize(relation:, cutoff_at:, byte_limit:, tmp_dir: Rails.root.join("tmp"), partition_column: nil, partition_value: nil) ⇒ ArchiveSerializer

relation: the exact records to serialize (callers pass a frozen id set, never a bare range), streamed in primary key order via find_each.

partition_column/partition_value describe the partition every record in this batch belongs to (see Raif.config.archive_partition_column) and are recorded in the manifest. Both fields are omitted entirely when partitioning is unset; a present partition_column with a nil partition_value marks an explicitly ungrouped batch.



38
39
40
41
42
43
44
45
 
# File 'app/models/raif/archive_serializer.rb', line 38

def initialize(relation:, cutoff_at:, byte_limit:, tmp_dir: Rails.root.join("tmp"), partition_column: nil, partition_value: nil)
  @relation = relation
  @cutoff_at = cutoff_at
  @byte_limit = byte_limit
  @tmp_dir = tmp_dir.to_s
  @partition_column = partition_column
  @partition_value = partition_value
end

Instance Method Details

#serialize ⇒ Object

=> { path:, checksum_sha256:, compressed_bytes:, record_ids: } The caller owns the file at path: and must delete it when done.



49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
 
# File 'app/models/raif/archive_serializer.rb', line 49

def serialize
  record_ids = []

  Tempfile.create(["raif-archive-spool", ".jsonl"], @tmp_dir) do |spool|
    spool.binmode
    uncompressed_bytes = 0

    @relation.find_each do |record|
      # to_json (not JSON.generate) so ActiveSupport encodes timestamps as
      # ISO8601 with sub-second precision; round-trip fidelity matters.
      line = "#{record.attributes.to_json}\n"
      spool.write(line)
      record_ids << record.id
      uncompressed_bytes += line.bytesize
      break if uncompressed_bytes >= @byte_limit
    end

    spool.flush
    spool.rewind

    path = write_gzip_file(spool, record_ids)

    {
      path: path,
      checksum_sha256: Digest::SHA256.file(path).hexdigest,
      compressed_bytes: File.size(path),
      record_ids: record_ids
    }
  end
end