Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object #57

pauldintel · 2023-07-18T23:28:45Z

I have launched the server using the docker image and vector DB launched fine :
Same machine in other session I am using the vector benchmark run . I am using python and pip3.10 .

[root@9049fa05600b ~]# docker run -p 6333:6333 qdrant/qdrant
_ _
__ _ | | __ __ _ _ __ | |
/ |/ _ | '__/ ` | ' | __|
| (| | (| | | | (| | | | | |_
_, |_,|| _,|| ||_|
||

Access web UI at http://localhost:6333/dashboard

[2023-07-18T22:34:56.204Z INFO storage::content_manager::consensus::persistent] Initializing new raft state at ./storage/raft_state
[2023-07-18T22:34:56.236Z INFO qdrant] Distributed mode disabled
[2023-07-18T22:34:56.236Z INFO qdrant] Telemetry reporting enabled, id: a96d4b56-dba5-4f4c-9332-04ab9c9033eb
[2023-07-18T22:34:56.238Z INFO qdrant::tonic] Qdrant gRPC listening on 6334
[2023-07-18T22:34:56.238Z INFO qdrant::tonic] TLS disabled for gRPC API
[2023-07-18T22:34:56.251Z INFO qdrant::actix] TLS disabled for REST API
[2023-07-18T22:34:56.251Z INFO qdrant::actix] Qdrant HTTP listening on 6333
[2023-07-18T22:34:56.251Z INFO actix_server::builder] Starting 7 workers
[2023-07-18T22:34:56.251Z INFO actix_server::server] Actix runtime found; starting in Actix runtime
[2023-07-18T22:35:04.526Z INFO actix_web::middleware::logger] 172.17.0.1 "DELETE /collections/benchmark HTTP/1.1" 200 72 "-" "python-httpx/0.24.1" 0.000457
[2023-07-18T22:35:04.530Z INFO actix_web::middleware::logger] 172.17.0.1 "DELETE /collections/benchmark HTTP/1.1" 200 69 "-" "python-httpx/0.24.1" 0.000119
[2023-07-18T22:35:05.562Z INFO actix_web::middleware::logger] 172.17.0.1 "PUT /collections/benchmark HTTP/1.1" 200 71 "-" "python-httpx/0.24.1" 1.031083

When I am running benchmark test getting below error :

[root@9049fa05600b vector-db-benchmark]# python3.10 -m run --engines -m-16- --datasets glove-*
Running experiment: qdrant-mmap-m-16-ef-128 - glove-25-angular
Downloading http://ann-benchmarks.com/glove-25-angular.hdf5...
Moving: /tmp/tmp6q2343mx -> /root/vectordb/vector-db-benchmark/datasets/glove-25-angular/glove-25-angular.hdf5
Experiment stage: Configure
Experiment stage: Upload
1343it [00:00, 18694.49it/s]
Experiment qdrant-mmap-m-16-ef-128 - glove-25-angular interrupted
Traceback (most recent call last):
File "/root/vectordb/vector-db-benchmark/run.py", line 52, in run
client.run_experiment(dataset, skip_upload, skip_search)
File "/root/vectordb/vector-db-benchmark/engine/base_client/client.py", line 70, in run_experiment
upload_stats = self.uploader.upload(
File "/root/vectordb/vector-db-benchmark/engine/base_client/upload.py", line 56, in upload
latencies = list(
File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 873, in next
raise value
multiprocessing.pool.MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x7f5fc488fe20>'. Reason: 'TypeError("cannot pickle '_thread.RLock' object")'
Traceback (most recent call last):

File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,

File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)

File "/root/vectordb/vector-db-benchmark/run.py", line 79, in
app()

File "/root/vectordb/vector-db-benchmark/run.py", line 74, in run
raise e

File "/root/vectordb/vector-db-benchmark/run.py", line 52, in run
client.run_experiment(dataset, skip_upload, skip_search)

File "/root/vectordb/vector-db-benchmark/engine/base_client/client.py", line 70, in run_experiment
upload_stats = self.uploader.upload(

File "/root/vectordb/vector-db-benchmark/engine/base_client/upload.py", line 56, in upload
latencies = list(

File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 873, in next
raise value

multiprocessing.pool.MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x7f5fc488fe20>'. Reason: 'TypeError("cannot pickle '_thread.RLock' object")'

Also there is no content generated inside results dir.
Any pointer how to resolve this issue ?
thanks

The text was updated successfully, but these errors were encountered:

chenbt-hz · 2023-07-21T08:36:49Z

I'm facing the same issue here. My system environment is CentOS Linux release 7.4.1708 (Core), and I'm using Python 3.8.1.

Have you found any solution or workaround for this problem?

pauldintel · 2023-07-21T16:20:29Z

I'm facing the same issue here. My system environment is CentOS Linux release 7.4.1708 (Core), and I'm using Python 3.8.1.

Have you found any solution or workaround for this problem?

I am using CentOS 8 stream and Python 3.10 ... no solution yet .waiting for the repo owners to comment .

chenbt-hz · 2023-07-24T08:13:20Z

I'm facing the same issue here. My system environment is CentOS Linux release 7.4.1708 (Core), and I'm using Python 3.8.1.
Have you found any solution or workaround for this problem?

I am using CentOS 8 stream and Python 3.10 ... no solution yet .waiting for the repo owners to comment .

Hello, I referred to the testing steps in this repository (https://github.com/myscale/vector-db-benchmark/blob/master/docs/step-by-step-guide-for-benchmark.md#qdrant) and made some minor modifications to the QdrantUploader.init_client method in qdrant/engine/clients/qdrant/upload.py.
I understand that the program is now able to run, but it appears to be hanging during runtime and there are anomalies detected in the collection status.
`
curl 'http://{ip}:{port}/collections/benchmark'

{"result":{"status":"red","optimizer_status":{"error":"Service internal error: IO Error: Function not implemented (os error 38)"},"vectors_count":12ctors_count":0,"points_count":1240620,"segments_count":24,"config":{"params":{"vectors":{"size":100,"distance":"Cosine"},"shard_number":3,"replicatite_consistency_factor":1,"on_disk_payload":true},"hnsw_config":{"m":16,"ef_construct":100,"full_scan_threshold":10000,"max_indexing_threads":0,"on_dizer_config":{"deleted_threshold":0.2,"vacuum_min_vector_number":1000,"default_segment_number":0,"max_segment_size":null,"memmap_threshold":10000,"i":20000,"flush_interval_sec":5,"max_optimization_threads":1},"wal_config":{"wal_capacity_mb":32,"wal_segments_ahead":0},"quantization_config":null},{}},"status":"ok","time":0.001639665}
`

qbx2 · 2023-11-24T06:43:06Z

You forgot to add -p 6334:6334

KShivendu · 2024-01-22T06:57:40Z

This was fixed in 9b22a84

Feel free to let us know if you continue to face the error.

gingerwizard · 2024-10-03T10:52:22Z

@KShivendu still facing this when running the qdrant tests, is a specific version of Python needed?

Traceback (most recent call last):

  File "<frozen runpy>", line 198, in _run_module_as_main

  File "<frozen runpy>", line 88, in _run_code

  File "/opt/vector-db-benchmark/run.py", line 91, in <module>
    app()

  File "/opt/vector-db-benchmark/run.py", line 86, in run
    raise e

  File "/opt/vector-db-benchmark/run.py", line 59, in run
    client.run_experiment(

  File "/opt/vector-db-benchmark/engine/base_client/client.py", line 115, in run_experiment
    upload_stats = self.uploader.upload(
                   ^^^^^^^^^^^^^^^^^^^^^

  File "/opt/vector-db-benchmark/engine/base_client/upload.py", line 56, in upload
    latencies = list(
                ^^^^^

  File "/home/ubuntu/.pyenv/versions/3.11.10/lib/python3.11/multiprocessing/pool.py", line 873, in next
    raise value

multiprocessing.pool.MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x710345e4a950>'. Reason: 'TypeError("cannot pickle '_thread.RLock' object")

glorinli · 2024-10-23T01:36:33Z

@KShivendu still facing this when running the qdrant tests, is a specific version of Python needed?

Traceback (most recent call last):

  File "<frozen runpy>", line 198, in _run_module_as_main

  File "<frozen runpy>", line 88, in _run_code

  File "/opt/vector-db-benchmark/run.py", line 91, in <module>
    app()

  File "/opt/vector-db-benchmark/run.py", line 86, in run
    raise e

  File "/opt/vector-db-benchmark/run.py", line 59, in run
    client.run_experiment(

  File "/opt/vector-db-benchmark/engine/base_client/client.py", line 115, in run_experiment
    upload_stats = self.uploader.upload(
                   ^^^^^^^^^^^^^^^^^^^^^

  File "/opt/vector-db-benchmark/engine/base_client/upload.py", line 56, in upload
    latencies = list(
                ^^^^^

  File "/home/ubuntu/.pyenv/versions/3.11.10/lib/python3.11/multiprocessing/pool.py", line 873, in next
    raise value

multiprocessing.pool.MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x710345e4a950>'. Reason: 'TypeError("cannot pickle '_thread.RLock' object")

Same issue

glorinli · 2024-10-23T03:10:03Z

In my case, the unpicklable is not in the passed-in parameters but seems the result, because I use wrong port.

KShivendu closed this as completed Jan 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object #57

Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object #57

pauldintel commented Jul 18, 2023

chenbt-hz commented Jul 21, 2023

pauldintel commented Jul 21, 2023

chenbt-hz commented Jul 24, 2023

qbx2 commented Nov 24, 2023

KShivendu commented Jan 22, 2024

gingerwizard commented Oct 3, 2024

glorinli commented Oct 23, 2024

glorinli commented Oct 23, 2024

Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object #57

Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object #57

Comments

pauldintel commented Jul 18, 2023

chenbt-hz commented Jul 21, 2023

pauldintel commented Jul 21, 2023

chenbt-hz commented Jul 24, 2023

qbx2 commented Nov 24, 2023

KShivendu commented Jan 22, 2024

gingerwizard commented Oct 3, 2024

glorinli commented Oct 23, 2024

glorinli commented Oct 23, 2024