feat: Parallelize uv install (windmill-labs#4774)

* Implement MVP of Parallel uv installation * Implement PY_CONCURRENT_DOWNLOADS * Remove Flock for uv installs * Make S3 pull/push parallel * Refactor and allow to Cancel installation * Dont print S3 in output if disabled * Implement better error handling * Polishing * More polishing * Implement error-handler for kill_tx_2.send() * Fix and Format prev merge * Presubscribe to all kill_tx's We do it now before first event could fire Meaning no events can be lost anymore * Early print errors and safer error handling * Return Err if installation failed * Final changes * Return error instead of just printing it * Safer the way to acquire permit * Fix compilation error * Remove double error logs
mastersatish · Nov 29, 2024 · 96371bc · 96371bc
1 parent 8ebb1fe
commit 96371bc
Show file tree

Hide file tree

Showing 3 changed files with 555 additions and 292 deletions.
diff --git a/README.md b/README.md
@@ -334,7 +334,8 @@ you to have it being synced automatically everyday.
 | GO_PATH                   | /usr/bin/go            | The path to the go binary.                                                                                                                                                                         | Worker                |
 | GOPRIVATE                 |                        | The GOPRIVATE env variable to use private go modules                                                                                                                                               | Worker                |
 | GOPROXY                   |                        | The GOPROXY env variable to use                                                                                                                                                                    | Worker                |
-| NETRC                     |                        | The netrc content to use a private go registry                                                                                                                                                     | Worker                |
+| NETRC                     |                        | The netrc content to use a private go registry                                                                                                                                                     | Worker                |                                                                                                                                                                  | Worker                |
+| PY_CONCURRENT_DOWNLOADS   | 20                     | Sets the maximum number of in-flight concurrent python downloads that windmill will perform at any given time.                                                                                     | Worker                |
 | PATH                      | None                   | The path environment variable, usually inherited                                                                                                                                                   | Worker                |
 | HOME                      | None                   | The home directory to use for Go and Bash , usually inherited                                                                                                                                      | Worker                |
 | DATABASE_CONNECTIONS      | 50 (Server)/3 (Worker) | The max number of connections in the database connection pool                                                                                                                                      | All                   |