Class: Raif::Llms::Bedrock
- Inherits:
-
Raif::Llm
- Object
- Raif::Llm
- Raif::Llms::Bedrock
- Includes:
- Concerns::Llms::Bedrock::MessageFormatting, Concerns::Llms::Bedrock::ResponseToolCalls, Concerns::Llms::Bedrock::ToolFormatting, Concerns::Llms::NativeJsonResponseValidation
- Defined in:
- app/models/raif/llms/bedrock.rb
Constant Summary collapse
- INFERENCE_PROFILE_PREFIXES =
Cross-region inference profile prefixes. A manifest api_name that already starts with one names its profile, as global-only models such as Claude Sonnet 5.5 must, so the configured prefix is not added.
%w[global us us-gov eu apac jp au ca].freeze
Class Method Summary collapse
- .cache_creation_input_token_cost_multiplier ⇒ Object
- .cache_read_input_token_cost_multiplier ⇒ Object
- .prompt_tokens_include_cached_tokens? ⇒ Boolean
Instance Method Summary collapse
Class Method Details
.cache_creation_input_token_cost_multiplier ⇒ Object
22 23 24 |
# File 'app/models/raif/llms/bedrock.rb', line 22 def self.cache_creation_input_token_cost_multiplier 1.25 end |
.cache_read_input_token_cost_multiplier ⇒ Object
18 19 20 |
# File 'app/models/raif/llms/bedrock.rb', line 18 def self.cache_read_input_token_cost_multiplier 0.1 end |
.prompt_tokens_include_cached_tokens? ⇒ Boolean
14 15 16 |
# File 'app/models/raif/llms/bedrock.rb', line 14 def self.prompt_tokens_include_cached_tokens? false end |
Instance Method Details
#perform_model_completion!(model_completion, &block) ⇒ Object
26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 |
# File 'app/models/raif/llms/bedrock.rb', line 26 def perform_model_completion!(model_completion, &block) model_completion.model_api_name = resolve_model_api_name(model_completion.model_api_name) params = build_request_parameters(model_completion) if model_completion.stream_response? bedrock_client.converse_stream(params) do |stream| stream.on_error_event do |event| raise Raif::Errors::StreamingError.new( message: event., type: event.event_type, code: event.error_code, event: event ) end handler = streaming_chunk_handler(model_completion, &block) stream.on_event do |event| handler.call(event) end end else response = bedrock_client.converse(params) update_model_completion(model_completion, response) end # Not in update_model_completion: a stream reports usage in a metadata event after # message_stop, and raising at message_stop would save the completion with zero tokens. validate_native_json_response!(model_completion) model_completion end |