feat(proxy_server.py): check if views exist on proxy server startup +… (

BerriAI#6360)

* feat(proxy_server.py): check if views exist on proxy server startup + refactor startup event logic to <50 LOC

* refactor(redis_cache.py): use a default cache value when writing to r… (BerriAI#6358)

* refactor(redis_cache.py): use a default cache value when writing to redis

prevent redis from blowing up in high traffic

* refactor(redis_cache.py): refactor all cache writes to use self.get_ttl

ensures default ttl always used when writing to redis

Prevents redis db from blowing up in prod

* feat(proxy_cli.py): add new 'log_config' cli param (BerriAI#6352)

* feat(proxy_cli.py): add new 'log_config' cli param

Allows passing logging.conf to uvicorn on startup

* docs(cli.md): add logging conf to uvicorn cli docs

* fix(get_llm_provider_logic.py): fix default api base for litellm_proxy

Fixes BerriAI#6332

* feat(openai_like/embedding): Add support for jina ai embeddings

Closes BerriAI#6337

* docs(deploy.md): update entrypoint.sh filepath post-refactor

Fixes outdated docs

* feat(prometheus.py): emit time_to_first_token metric on prometheus

Closes BerriAI#6334

* fix(prometheus.py): only emit time to first token metric if stream is True

enables more accurate ttft usage

* test: handle vertex api instability

* fix(get_llm_provider_logic.py): fix import

* fix(openai.py): fix deepinfra default api base

* fix(anthropic/transformation.py): remove anthropic beta header (BerriAI#6361)

* docs(sidebars.js): add jina ai embedding to docs

* docs(sidebars.js): add jina ai to left nav

* bump: version 1.50.1 → 1.50.2

* langfuse use helper for get_langfuse_logging_config

* Refactor: apply early return (BerriAI#6369)

* (refactor) remove berrispendLogger - unused logging integration  (BerriAI#6363)

* fix remove berrispendLogger

* remove unused clickhouse logger

* fix docs configs.md

* (fix) standard logging metadata + add unit testing  (BerriAI#6366)

* fix setting StandardLoggingMetadata

* add unit testing for standard logging metadata

* fix otel logging test

* fix linting

* fix typing

* Revert "(fix) standard logging metadata + add unit testing  (BerriAI#6366)" (BerriAI#6381)

This reverts commit 8359cb6.

* add new 35 mode lcard (BerriAI#6378)

* Add claude 3 5 sonnet 20241022 models for all provides (BerriAI#6380)

* Add Claude 3.5 v2 on Amazon Bedrock and Vertex AI.

* added anthropic/claude-3-5-sonnet-20241022

* add new 35 mode lcard

---------

Co-authored-by: Paul Gauthier <[email protected]>
Co-authored-by: lowjiansheng <[email protected]>

* test(skip-flaky-google-context-caching-test): google is not reliable. their sample code is also not working

* test(test_alangfuse.py): handle flaky langfuse test better

* (feat) Arize - Allow using Arize HTTP endpoint  (BerriAI#6364)

* arize use helper for get_arize_opentelemetry_config

* use helper to get Arize OTEL config

* arize add helpers for arize

* docs allow using arize http endpoint

* fix importing OTEL for Arize

* use static methods for ArizeLogger

* fix ArizeLogger tests

* Litellm dev 10 22 2024 (BerriAI#6384)

* fix(utils.py): add 'disallowed_special' for token counting on .encode()

Fixes error when '<
endoftext
>' in string

* Revert "(fix) standard logging metadata + add unit testing  (BerriAI#6366)" (BerriAI#6381)

This reverts commit 8359cb6.

* add new 35 mode lcard (BerriAI#6378)

* Add claude 3 5 sonnet 20241022 models for all provides (BerriAI#6380)

* Add Claude 3.5 v2 on Amazon Bedrock and Vertex AI.

* added anthropic/claude-3-5-sonnet-20241022

* add new 35 mode lcard

---------

Co-authored-by: Paul Gauthier <[email protected]>
Co-authored-by: lowjiansheng <[email protected]>

* test(skip-flaky-google-context-caching-test): google is not reliable. their sample code is also not working

* Fix metadata being overwritten in speech() (BerriAI#6295)

* fix: adding missing redis cluster kwargs (BerriAI#6318)

Co-authored-by: Ali Arian <[email protected]>

* Add support for `max_completion_tokens` in Azure OpenAI (BerriAI#6376)

Now that Azure supports `max_completion_tokens`, no need for special handling for this param and let it pass thru. More details: https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models?tabs=python-secure#api-support

* build(model_prices_and_context_window.json): add voyage-finance-2 pricing

Closes BerriAI#6371

* build(model_prices_and_context_window.json): fix llama3.1 pricing model name on map

Closes BerriAI#6310

* feat(realtime_streaming.py): just log specific events

Closes BerriAI#6267

* fix(utils.py): more robust checking if unmapped vertex anthropic model belongs to that family of models

Fixes BerriAI#6383

* Fix Ollama stream handling for tool calls with None content (BerriAI#6155)

* test(test_max_completions): update test now that azure supports 'max_completion_tokens'

* fix(handler.py): fix linting error

---------

Co-authored-by: Ishaan Jaff <[email protected]>
Co-authored-by: Low Jian Sheng <[email protected]>
Co-authored-by: David Manouchehri <[email protected]>
Co-authored-by: Paul Gauthier <[email protected]>
Co-authored-by: John HU <[email protected]>
Co-authored-by: Ali Arian <[email protected]>
Co-authored-by: Ali Arian <[email protected]>
Co-authored-by: Anand Taralika <[email protected]>
Co-authored-by: Nolan Tremelling <[email protected]>

* bump: version 1.50.2 → 1.50.3

* build(deps): bump http-proxy-middleware in /docs/my-website (BerriAI#6395)

Bumps [http-proxy-middleware](https://github.com/chimurai/http-proxy-middleware) from 2.0.6 to 2.0.7.
- [Release notes](https://github.com/chimurai/http-proxy-middleware/releases)
- [Changelog](https://github.com/chimurai/http-proxy-middleware/blob/v2.0.7/CHANGELOG.md)
- [Commits](chimurai/http-proxy-middleware@v2.0.6...v2.0.7)

---
updated-dependencies:
- dependency-name: http-proxy-middleware
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* (docs + testing) Correctly document the timeout value used by litellm proxy is 6000 seconds + add to best practices for prod  (BerriAI#6339)

* fix docs use documented timeout

* document request timeout

* add test for litellm.request_timeout

* add test for checking value of timeout

* (refactor) move convert dict to model response to llm_response_utils/ (BerriAI#6393)

* refactor move convert dict to model response

* fix imports

* fix import _handle_invalid_parallel_tool_calls

* (refactor) litellm.Router client initialization utils  (BerriAI#6394)

* refactor InitalizeOpenAISDKClient

* use helper func for _should_create_openai_sdk_client_for_model

* use static methods for set client on litellm router

* reduce LOC in _get_client_initialization_params

* fix _should_create_openai_sdk_client_for_model

* code quality fix

* test test_should_create_openai_sdk_client_for_model

* test test_get_client_initialization_params_openai

* fix mypy linting errors

* fix OpenAISDKClientInitializationParams

* test_get_client_initialization_params_all_env_vars

* test_get_client_initialization_params_azure_ai_studio_mistral

* test_get_client_initialization_params_default_values

* fix _get_client_initialization_params

* (fix) Langfuse key based logging  (BerriAI#6372)

* langfuse use helper for get_langfuse_logging_config

* fix get_langfuse_logger_for_request

* fix import

* fix get_langfuse_logger_for_request

* test_get_langfuse_logger_for_request_with_dynamic_params

* unit testing for test_get_langfuse_logger_for_request_with_no_dynamic_params

* parameterized langfuse testing

* fix langfuse test

* fix langfuse logging

* fix test_aaalangfuse_logging_metadata

* fix langfuse log metadata test

* fix langfuse logger

* use create_langfuse_logger_from_credentials

* fix test_get_langfuse_logger_for_request_with_no_dynamic_params

* fix correct langfuse/ folder structure

* use static methods for langfuse logger

* add commment on langfuse handler

* fix linting error

* add unit testing for langfuse logging

* fix linting

* fix failure handler langfuse

* Revert "(refactor) litellm.Router client initialization utils  (BerriAI#6394)" (BerriAI#6403)

This reverts commit b70147f.

* def test_text_completion_with_echo(stream): (BerriAI#6401)

test

* fix linting - remove # noqa PLR0915 from fixed function

* test: cleanup codestral tests - backend api unavailable

* (refactor) prometheus async_log_success_event to be under 100 LOC  (BerriAI#6416)

* unit testig for prometheus

* unit testing for success metrics

* use 1 helper for _increment_token_metrics

* use helper for _increment_remaining_budget_metrics

* use _increment_remaining_budget_metrics

* use _increment_top_level_request_and_spend_metrics

* use helper for _set_latency_metrics

* remove noqa violation

* fix test prometheus

* test prometheus

* unit testing for all prometheus helper functions

* fix prom unit tests

* fix unit tests prometheus

* fix unit test prom

* (refactor) router - use static methods for client init utils  (BerriAI#6420)

* use InitalizeOpenAISDKClient

* use InitalizeOpenAISDKClient static method

* fix  # noqa: PLR0915

* (code cleanup) remove unused and undocumented logging integrations - litedebugger, berrispend  (BerriAI#6406)

* code cleanup remove unused and undocumented code files

* fix unused logging integrations cleanup

* bump: version 1.50.3 → 1.50.4

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Ishaan Jaff <[email protected]>
Co-authored-by: Hakan Taşköprü <[email protected]>
Co-authored-by: Low Jian Sheng <[email protected]>
Co-authored-by: David Manouchehri <[email protected]>
Co-authored-by: Paul Gauthier <[email protected]>
Co-authored-by: John HU <[email protected]>
Co-authored-by: Ali Arian <[email protected]>
Co-authored-by: Ali Arian <[email protected]>
Co-authored-by: Anand Taralika <[email protected]>
Co-authored-by: Nolan Tremelling <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

litellm/integrations/prometheus.py

@@ -346,12 +346,14 @@ async def async_log_success_event(self, kwargs, response_obj, start_time, end_ti
         standard_logging_payload: Optional[StandardLoggingPayload] = kwargs.get(
             "standard_logging_object"
         )
+
         if standard_logging_payload is None or not isinstance(
             standard_logging_payload, dict
         ):
             raise ValueError(
                 f"standard_logging_object is required, got={standard_logging_payload}"
             )
+
         model = kwargs.get("model", "")
         litellm_params = kwargs.get("litellm_params", {}) or {}
         _metadata = litellm_params.get("metadata", {})

litellm/litellm_core_utils/llm_response_utils/convert_dict_to_response.py

@@ -260,6 +260,7 @@ def convert_to_model_response_object(  # noqa: PLR0915
     ] = None,  # used for supporting 'json_schema' on older models
 ):
     received_args = locals()
+
     additional_headers = get_response_headers(_response_headers)
 
     if hidden_params is None:
@@ -448,11 +449,13 @@ def convert_to_model_response_object(  # noqa: PLR0915
         ):
             if response_object is None:
                 raise Exception("Error in response object format")
+
             return LiteLLMResponseObjectHandler.convert_to_image_response(
                 response_object=response_object,
                 model_response_object=model_response_object,
                 hidden_params=hidden_params,
             )
+
         elif response_type == "audio_transcription" and (
             model_response_object is None
             or isinstance(model_response_object, TranscriptionResponse)

litellm/proxy/_new_secret_config.yaml

@@ -48,4 +48,3 @@ router_settings:
   redis_host: os.environ/REDIS_HOST
   redis_port: os.environ/REDIS_PORT
   redis_password: os.environ/REDIS_PASSWORD
-