Max retries exceeded with url: /v1/embeddings

[Thuandv-seta]

Self Checks

• This is only for bug report, if you would like to ask a question, please head to Discussions.
• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• Please do not modify this template :) and fill in all the required fields.

Dify version

latest

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

1. I run embedding model with vllm 0.6.6

CUDA_VISIBLE_DEVICES=0,1 vllm serve ../llm_models/KaLM-embedding-multilingual-mini-instruct-v1.5 \
    --task embedding \
    --host 0.0.0.0 \
    --port 1724 \
    --served-model-name KaLM-embedding \
    --dtype half \
    --api-key token-abc1234 \
    --tensor-parallel-size 2

2. I follow the steps

cd dify
cd docker
cp .env.example .env
docker-compose up -d

3. Add model in model provider, select OpenAI-API-compatible
   

4. If I serve model with port 8000, everything works fine. But I want to serve few models with vllm using different ports.

✔️ Expected Behavior

I can add many models successfully

❌ Actual Behavior

Traceback (most recent call last):
  File "/app/api/controllers/console/workspace/models.py", line 154, in post
    model_provider_service.save_model_credentials(
  File "/app/api/services/model_provider_service.py", line 242, in save_model_credentials
    provider_configuration.add_or_update_custom_model_credentials(
  File "/app/api/core/entities/provider_configuration.py", line 393, in add_or_update_custom_model_credentials
    provider_model_record, credentials = self.custom_model_credentials_validate(model_type, model, credentials)
                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/entities/provider_configuration.py", line 373, in custom_model_credentials_validate
    credentials = model_provider_factory.model_credentials_validate(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/model_provider_factory.py", line 123, in model_credentials_validate
    model_instance.validate_credentials(model, filtered_credentials)
  File "/app/api/core/model_runtime/model_providers/openai_api_compatible/text_embedding/text_embedding.py", line 172, in validate_credentials
    raise CredentialsValidateFailedError(str(ex))
core.model_runtime.errors.validate.CredentialsValidateFailedError: HTTPConnectionPool(host='192.168.80.25', port=1724): Max retries exceeded with url: /v1/embeddings (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f0046241d00>, 'Connection to 192.168.80.25 timed out. (connect timeout=10)'))

[dosubot]

@Thuandv-seta Hello there! 👋 I'm here to help you with your bug report. I can assist with solving bugs, answering questions, and even guide you on how to become a contributor. Let's squash those bugs together!

I found two similar unsolved discussions that might be relevant to your issue:

1. An error occurred during credentials validation ^[1].
2. Connection error when set up model provider ^[2].

Regarding your issue, the error message indicates a ConnectTimeoutError, which suggests that the connection to the specified host and port is timing out. Here are a few things you might consider checking:

1. Network Configuration: Ensure that the network settings allow connections to the specified host and port. Verify that there are no firewall rules or network policies blocking the connection.

2. Port Availability: Since serving the model on port 8000 works fine, ensure that the port 1724 is open and not being used by another service.

3. Timeout Settings: You might want to adjust the timeout settings to allow more time for the connection to be established. Dify allows configuration of HTTP connection settings, including timeouts, through environment variables. You can set these in your .env file:

   • HTTP_REQUEST_MAX_CONNECT_TIMEOUT
   • HTTP_REQUEST_MAX_READ_TIMEOUT
   • HTTP_REQUEST_MAX_WRITE_TIMEOUT

   For example, increasing the HTTP_REQUEST_MAX_CONNECT_TIMEOUT might help if the connection is taking longer than expected to establish ^[3].

4. Endpoint Configuration: Double-check the API endpoint URL configuration to ensure it is correct and accessible from the environment where Dify is running.

These steps should help you troubleshoot the connection timeout issue.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[crazywoola]

I don't think this is related to dify itself. If you can connect via default config then it should work with other ports. I suggest you checking the configuration of the provider itself.