🌐 Web query:

litellm completion_cost model parameter override when completion_response has different model

💡 Result:

To handle cases where you need to override the model for cost calculation in LiteLLM (for example, when a completion response returns a model name that differs from the one you want to bill against), you should use the base_model parameter [1]. When using the LiteLLM proxy or configuration, you can set the base_model in your model_list configuration. This tells LiteLLM to use the specified base_model for pricing lookups instead of the model name returned in the completion response [1]. If you are calling the completion_cost function directly in your code, you can explicitly pass the desired model name via the model argument to ensure the cost is calculated against that specific pricing tier [2][3]. For more advanced scenarios involving custom pricing, you can: 1. Override specific cost parameters (e.g., input_cost_per_token, output_cost_per_token) within the model_info block of your configuration [4][1]. 2. Use register_model to manually define or override model pricing in the global LiteLLM cost dictionary [2]. 3. For proxy callbacks, it is often more reliable to use the pre-computed response_cost found in the kwargs of the response object rather than manually invoking completion_cost, as this value is typically calculated by LiteLLM using the intended model context before aliasing occurs [5]. When overriding, note that setting custom pricing fields detaches the deployment from the default cost map for that entry [4]. Ensure that you define all necessary fields, as partial overrides can lead to unexpected billing results [4].

Citations:

• 1: https://docs.litellm.ai/docs/proxy/custom_pricing
• 2: https://docs.litellm.ai/docs/completion/token_usage
• 3: https://github.com/BerriAI/litellm/blob/62920a0c/litellm/cost_calculator.py
• 4: https://docs.litellm.ai/docs/proxy/custom_model_cost_map
• 5: [Bug]: completion_cost function broken when used in a custom proxy callback in recent versions BerriAI/litellm#22551

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== locate llm.py and relevant symbols =="
fd -a 'llm\.py$' . | sed 's#^\./##'
echo
rg -n "def _compute_call_cost|_cost_model|cost_model|AWSBedrockLLMParameters|model_id|litellm\.completion_cost|litellm\.cost_per_token" -S .

echo
echo "== inspect llm.py relevant sections =="
LLM_FILES="$(fd 'llm.py$' .)"
for f in $LLM_FILES; do
  echo "--- $f ($(wc -l < "$f") lines) ---"
  rg -n "_compute_call_cost|_record_usage|_cost_model|cost_model|AWSBedrockLLMParameters|model_id|completion_cost|cost_per_token" -C 4 "$f"
done

echo
echo "== inspect litellm installed versions/docs signatures if available =="
python3 - <<'PY'
import importlib.util
spec = importlib.util.find_spec("litellm")
print("litellm_module_path:", getattr(spec, "origin", None))
if spec:
    import litellm
    import inspect
    print("litellm_version:", getattr(litellm, "__version__", None))
    for name in ("completion_cost", "cost_per_token"):
        fn = getattr(litellm, name, None)
        print(f"litellm.{name}:", fn)
        if fn is not None:
            try:
                print(inspect.signature(fn))
            except Exception as e:
                print("signature_error:", e)
            try:
                src = inspect.getsource(fn)
                lines = src.splitlines()
                for i, line in enumerate(lines[:40], 1):
                    print(f"{i}:{line}")
            except Exception:
                pass
PY

Repository: Zipstack/unstract

Length of output: 11605

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== locate llm.py and relevant symbols =="
fd -a 'llm\.py$' . | sed 's#^\./##'
echo
rg -n "def _compute_call_cost|_cost_model|cost_model|AWSBedrockLLMParameters|model_id|litellm\.completion_cost|litellm\.cost_per_token" -S .

echo
echo "== inspect llm.py relevant sections =="
LLM_FILES="$(fd 'llm.py$' .)"
for f in $LLM_FILES; do
  echo "--- $f ($(wc -l < "$f") lines) ---"
  rg -n "_compute_call_cost|_record_usage|_cost_model|cost_model|AWSBedrockLLMParameters|model_id|completion_cost|cost_per_token" -C 4 "$f"
done

echo
echo "== inspect litellm installed versions/docs signatures if available =="
python3 - <<'PY'
import importlib.util
spec = importlib.util.find_spec("litellm")
print("litellm_module_path:", getattr(spec, "origin", None))
if spec:
    import litellm
    import inspect
    print("litellm_version:", getattr(litellm, "__version__", None))
    for name in ("completion_cost", "cost_per_token"):
        fn = getattr(litellm, name, None)
        print(f"litellm.{name}:", fn)
        if fn is not None:
            try:
                print(inspect.signature(fn))
            except Exception as e:
                print("signature_error:", e)
            try:
                src = inspect.getsource(fn)
                lines = src.splitlines()
                for i, line in enumerate(lines[:80], 1):
                    print(f"{i}:{line}")
            except Exception:
                pass
PY

Repository: Zipstack/unstract

Length of output: 11605

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== llm.py cost flow section =="
sed -n '220,260p' unstract/sdk1/src/unstract/sdk1/llm.py
sed -n '420,470p' unstract/sdk1/src/unstract/sdk1/llm.py
sed -n '540,575p' unstract/sdk1/src/unstract/sdk1/llm.py
sed -n '635,655p' unstract/sdk1/src/unstract/sdk1/llm.py
sed -n '700,715p' unstract/sdk1/src/unstract/sdk1/llm.py
sed -n '820,865p' unstract/sdk1/src/unstract/sdk1/llm.py

echo
echo "== base1 cost_model derivation context =="
sed -n '470,510p' unstract/sdk1/src/unstract/sdk1/adapters/base1.py
sed -n '985,1100p' unstract/sdk1/src/unstract/sdk1/adapters/base1.py
sed -n '1525,1545p' unstract/sdk1/src/unstract/sdk1/adapters/base1.py

echo
echo "== prompt caching tests =="
sed -n '1,140p' unstract/sdk1/tests/test_prompt_caching.py

echo
echo "== litellm completion source if available =="
python3 - <<'PY'
import importlib.util
spec = importlib.util.find_spec("litellm")
print("litellm_module_path:", getattr(spec, "origin", None))
if spec:
    import litellm
    import inspect
    print("litellm_version:", getattr(litellm, "__version__", None))
    for name in ("completion_cost", "cost_per_token"):
        fn = getattr(litellm, name, None)
        print(f"litellm.{name}:", fn)
        if fn is not None:
            try:
                print(inspect.signature(fn))
            except Exception as e:
                print("signature_error:", e)
            try:
                src = inspect.getsource(fn)
                for i, line in enumerate(src.splitlines()[:120], 1):
                    print(f"{i}:{line}")
                print("---")
            except Exception as e:
                print("source_error:", e)
PY

Repository: Zipstack/unstract

Length of output: 21711

🌐 Web query:

site:github.com/BerriAI/litellm completion_cost model param response model cost_per_token model

💡 Result:

The cost of an LLM completion request is typically calculated by summing the costs of input (prompt) and output (completion) tokens, often adjusted for factors like prompt caching or specialized modalities (e.g., audio) [1][2][3]. The standard formula for calculating the total cost is: Total Cost = (Input Tokens * Input Cost per Token) + (Output Tokens * Output Cost per Token) [4][3] Technical implementation details often include the following components: 1. Token Counts: The system calculates or extracts the number of prompt tokens and completion tokens generated during the API call [4][5]. 2. Unit Cost Mapping: Costs are determined by referencing a model-specific map that defines the price per unit (e.g., per 1 million tokens or per single token) for input and output [4][6]. 3. Advanced Cost Factors: - Prompt Caching: Costs may be further broken down into distinct rates for cache creation and cache reads [6][3]. - Multi-modal Adjustments: For models processing audio or images, additional costs per audio/image token may be added to the standard input/output calculations [1][2][7]. - Threshold Pricing: Some models apply different rates for tokens once usage exceeds specific thresholds (e.g., beyond 200k tokens) [6][2]. - Alternative Metrics: Some providers or specialized workloads (e.g., certain deployments on AWS Sagemaker or Replicate) may use "cost per second" based on inference runtime rather than token counts [6][5]. In production environments, developers often use middleware or helper utilities to automate these calculations by querying these model-specific price maps based on the provider and model name [4][6][3].

Citations:

• 1: https://github.com/BerriAI/litellm/blob/a05a1eef/litellm/llms/openai/cost_calculation.py
• 2: https://github.com/BerriAI/litellm/blob/84c1414a/litellm/litellm_core_utils/llm_cost_calc/utils.py
• 3: https://github.com/BerriAI/litellm/blob/f7ce1173f3315cc6cae06cf9bcf12e54a2a19705/litellm/llms/anthropic/cost_calculation.py
• 4: https://github.com/BerriAI/litellm/blob/main/docs/my-website/src/pages/token_usage.md
• 5: https://github.com/BerriAI/litellm/blob/a05a1eef/litellm/cost_calculator.py
• 6: https://github.com/BerriAI/litellm/blob/main/docs/my-website/docs/proxy/custom_pricing.md
• 7: https://github.com/BerriAI/litellm/blob/4dc9726d/litellm/llms/openai/cost_calculation.py

🌐 Web query:

"def completion_cost" "model =" "completion_response" litellm

💡 Result:

The completion_cost function in LiteLLM is used to calculate the cost (in USD) of a completion request [1][2]. To use it, you pass your litellm.completion() response object to the completion_response parameter [1][2]. Example usage: from litellm import completion, completion_cost # Make the completion call response = completion( model="gpt-4o", messages=[{"role": "user", "content": "Hello!"}]) # Calculate the cost cost = completion_cost(completion_response=response) print(f"Total cost: ${float(cost):.10f}") Key details: - Alternative inputs: If you do not have a response object, you can provide the model name along with prompt and completion strings to estimate the cost [1][2]. - Pre-computed cost: LiteLLM automatically computes the cost for every completion call and stores it in the response object [1]. You can often access this directly via response._hidden_params.get("response_cost") [1][3] or kwargs.get("response_cost") within custom logging callbacks [4]. - Customization: The function supports optional parameters for custom pricing, such as custom_cost_per_token or custom_llm_provider, to handle specialized models or scenarios [5][6][2].

Citations:

• 1: https://docs.litellm.ai/docs/completion/token_usage
• 2: https://github.com/BerriAI/litellm/blob/84c1414a/litellm/cost_calculator.py
• 3: Get response_cost BerriAI/litellm#22024
• 4: [Bug]: completion_cost function broken when used in a custom proxy callback in recent versions BerriAI/litellm#22551
• 5: https://github.com/BerriAI/litellm/blob/62920a0c/litellm/cost_calculator.py
• 6: https://github.com/BerriAI/litellm/blob/e15b37a1/tests/local_testing/test_completion_cost.py

🏁 Script executed:

#!/bin/bash
set -euo pipefail

if [ -n "$(command -v gh 2>/dev/null)" ]; then
  echo "== fetch LiteLLM cost calculator source from main =="
  gh api raw https://raw.githubusercontent.com/BerriAI/litellm/main/litellm/cost_calculator.py | sed -n '1,240p'
else
  echo "gh unavailable"
fi

echo
echo "== inspect LLM._record_usage =="
sed -n '750,832p' unstract/sdk1/src/unstract/sdk1/llm.py

echo
echo "== inspect cost_per_token implementation from installed package if available by file walk =="
python3 - <<'PY'
import importlib.util, sys, inspect
spec = importlib.util.find_spec("litellm")
if not spec:
    print("no literm installed")
    sys.exit(0)
print(spec.origin)
from pathlib import Path
base = Path(spec.submodule_search_locations[0]) if spec.submodule_search_locations else Path(spec.origin).parent
print("base", base)
files = list(base.rglob("*.py"))
for f in files[:20]:
    print(f)
PY

Repository: Zipstack/unstract

Length of output: 237

🌐 Web query:

raw.githubusercontent.com BerriAI litellm main litellm cost_calculator.py "model =" "completion_response"

💡 Result:

The file litellm/cost_calculator.py within the BerriAI/litellm repository is a core utility module used to calculate the cost of LLM API calls [1]. The mention of completion_response in your query refers to the parameter used by functions in this module—specifically completion_cost—to extract usage information (such as prompt and completion tokens) and the model name to compute the total cost of a response [1]. Key details regarding this file and its usage include: 1. Functionality: The completion_cost function calculates the total cost (in USD) for a given LLM response object [2][1]. If the completion_response is provided, it automatically extracts token counts and the model name to perform the calculation [1]. If not provided, it can calculate costs based on provided model names and text inputs [1]. 2. Dependency: It relies on model pricing information, which LiteLLM typically fetches from a JSON file (model_prices_and_context_window.json) [2][1]. 3. Usage Pattern: Developers typically use it as follows: from litellm import completion_cost cost = completion_cost(completion_response=response_object) 4. Integration: The logic in cost_calculator.py is also used by other LiteLLM components, such as the BudgetManager, to track and update user or model-specific costs [3]. If you are encountering issues with this file, note that it is frequently updated to support new models and providers, and it is standard practice to ensure your LiteLLM version is up to date to maintain accurate cost tracking [4].

Citations:

• 1: https://github.com/BerriAI/litellm/blob/62920a0c/litellm/cost_calculator.py
• 2: https://docs.litellm.ai/docs/completion/token_usage
• 3: https://github.com/BerriAI/litellm/blob/main/litellm/budget_manager.py
• 4: [Bug]:UnicodeDecodeError: 'charmap' on Windows during litellm import BerriAI/litellm#10272

🌐 Web query:

github.com/BerriAI/litellm/blob/main/litellm/cost_calculator.py "if completion_response is not None"

💡 Result:

In the LiteLLM library, the check if completion_response is not None within litellm/cost_calculator.py is used to determine how to proceed with calculating the cost of an LLM API call [1][2]. When completion_response is provided (is not None), the cost calculation logic extracts necessary information—specifically the model name and token usage—directly from the response object [1][2]. This allows LiteLLM to dynamically identify the specific model used for the completion, which is particularly useful in scenarios like dynamic routing (e.g., via Azure Model Router), where the actual model utilized may differ from the initial request model [3]. The code supports both Pydantic BaseModel objects and dictionaries as valid completion_response types, using getattr or dictionary access to retrieve the 'model' field and usage statistics [1]. If completion_response is None, the function falls back to calculating costs based on provided prompt and completion text strings, or other parameters like prompt_tokens and completion_tokens if they were passed separately [2][4][5]. In summary, this conditional check enables the library to handle both post-completion cost tracking (where a response object is available) and pre-completion cost estimation (where only input/output text or token counts are available) [2][5].

Citations:

• 1: https://github.com/BerriAI/litellm/blob/e15b37a1/litellm/cost_calculator.py
• 2: https://github.com/BerriAI/litellm/blob/62920a0c/litellm/cost_calculator.py
• 3: https://github.com/BerriAI/litellm/blob/62920a0c/tests/test_litellm/test_cost_calculator.py
• 4: https://docs.litellm.ai/docs/completion/token_usage
• 5: https://www.datacamp.com/tutorial/litellm

🌐 Web query:

raw.githubusercontent.com/BerriAI/litellm/main/litellm/cost_calculator.py "completion_cost"

💡 Result:

The completion_cost function in the LiteLLM library is a utility designed to calculate the total cost in USD for a given language model API call [1][2][3]. It is located in the litellm/cost_calculator.py module [1][4]. Key features and usage details include: 1. Functionality: It computes the total cost by combining token counts (for input/prompt and output/completion) with model-specific pricing data [2][3][5]. It utilizes a built-in model cost map [2][5]. 2. Primary Inputs: - completion_response: The response object returned from a LiteLLM completion call [1][3]. - Alternatively, users can provide the model name, the prompt (as a string), and the completion (as a string) to calculate costs manually without a full response object [1][2][3]. 3. Advanced Parameters: - Custom Pricing: Supports overriding default costs using parameters such as custom_cost_per_token or custom_cost_per_second, which is particularly useful for providers like Azure or SageMaker [1][4][6]. - Regional Pricing: Includes parameters like region_name for services with location-based pricing, such as Bedrock [1][4]. - Other: Accepts optional parameters for image generation (size, quality, n), service tiers, and specific routing identifiers [1][4]. 4. Return Value: The function returns a float representing the total USD cost for the specified interaction [1][3]. For implementation examples, users typically pass the response object from a litellm.completion call directly into completion_cost [7][3]. Alternatively, manual calculation can be performed by providing the model, prompt, and completion text directly to the function [2][3][5].

Citations:

• 1: https://github.com/BerriAI/litellm/blob/e15b37a1/litellm/cost_calculator.py
• 2: https://github.com/BerriAI/litellm/blob/main/docs/my-website/src/pages/token_usage.md
• 3: https://docs.litellm.ai/docs/completion/token_usage
• 4: https://github.com/BerriAI/litellm/blob/62920a0c/litellm/cost_calculator.py
• 5: https://docs.litellm.ai/token_usage
• 6: https://docs.litellm.ai/docs/sdk_custom_pricing
• 7: https://github.com/BerriAI/litellm/blob/main/cookbook/LiteLLM_Completion_Cost.ipynb

🏁 Script executed:

#!/bin/bash
set -euo pipefail

URL="https://raw.githubusercontent.com/BerriAI/litellm/main/litellm/cost_calculator.py"
tmp="$(mktemp)"

if command -v curl >/dev/null 2>&1; then
  curl -fsSL "$URL" -o "$tmp"
else
  python3 - <<'PY' - "$URL" "$tmp"
import urllib.request, sys
urllib.request.urlretrieve(sys.argv[1], sys.argv[2])
PY
fi

echo "== LiteLLM cost calculator snippet =="
sed -n '1,240p' "$tmp"

echo
echo "== model/model_name handling snippets =="
rg -n "model ?=|model_name|completion_response|usage|response_cost|cost_per_token|base_model" "$tmp" -C 3

rm -f "$tmp"

echo
echo "== inspect LLM._record_usage and cost_model caller context =="
sed -n '740,828p' unstract/sdk1/src/unstract/sdk1/llm.py

Repository: Zipstack/unstract

Length of output: 50373

🏁 Script executed:

#!/bin/bash
set -euo pipefail

URL="https://raw.githubusercontent.com/BerriAI/litellm/main/litellm/cost_calculator.py"
tmp="$(mktemp)"
curl -fsSL "$URL" -o "$tmp"

echo "== LiteLLM completion_price_model selection after model selection =="
sed -n '1100,1240p' "$tmp"

echo
echo "== LiteLLM usage/cost calculation calls for Bedrock route =="
sed -n '1600,1900p' "$tmp"

echo
echo "== LiteLLM generic_cost_per_token signature/body =="
sed -n '290,370p' "$tmp"

rm -f "$tmp"

Repository: Zipstack/unstract

Length of output: 23019

🌐 Web query:

github.com/BerriAI/litellm BedRockCostConfig cost_per_token anthropic models

💡 Result:

In LiteLLM, cost calculation for Anthropic models hosted on AWS Bedrock is handled primarily through the cost_per_token function located in litellm/llms/bedrock/cost_calculation.py [1]. This utility calculates costs by utilizing the same logic as standard Anthropic cost calculations, while ensuring the provider is identified as bedrock to facilitate appropriate pricing lookups [1]. To manage or override pricing for these models, LiteLLM provides a custom pricing mechanism [2]. You can override the default cost map by adding a model_info key to your model configuration in the LiteLLM proxy or config file [2]. Example configuration for overriding pricing: model_list: - model_name: "my-bedrock-claude" litellm_params: model: "bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0" model_info: input_cost_per_token: 0.000003 output_cost_per_token: 0.000015 For providers like Bedrock that support multiple pricing tiers (e.g., PayGo, priority, or flex), LiteLLM automatically selects the appropriate pricing based on the serviceTier returned in the response [2]. Note that there are known edge cases regarding cost calculation: 1. Cross-region inference: Some users have reported issues where region-specific pricing entries (e.g., for us.anthropic.claude-sonnet-4-6) are not correctly matched due to the model name stripping logic, which may default back to the base model's pricing instead [3]. 2. Provider-specific dispatch: In some configurations, if a custom_llm_provider is explicitly set, the cost calculator may dispatch to provider-specific functions (like anthropic_cost_per_token) before checking for custom override values, which can lead to custom zero-cost overrides being ignored [4].

Citations:

• 1: https://github.com/BerriAI/litellm/blob/d251238b/litellm/llms/bedrock/cost_calculation.py
• 2: https://docs.litellm.ai/docs/proxy/custom_pricing
• 3: [Bug]: Bedrock cross-region inference pricing entries in cost map are unreachable due to lookup order BerriAI/litellm#24669
• 4: [Bug]: custom_llm_provider "anthropic" bypasses input_cost_per_token: 0 — cost_per_token() dispatches to anthropic_cost_per_token() before checking custom pricing BerriAI/litellm#25204

• workers/executor/executors/answer_prompt.py#L163-L187 (this comment)
• workers/tests/test_answer_prompt_caching.py#L52-L91