Ability to split model components to different backend devices #461
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Green-Sky
suggested changes
Nov 14, 2024
stable-diffusion.cpp
Outdated
| } | ||
|
|
||
| } | ||
stable-diffusion.cpp
Outdated
| if (model_backend_index == -1) { | ||
| // default behavior, last device selected | ||
| for (int device = 0; device < ggml_backend_vk_get_device_count(); ++device) { | ||
| backend = ggml_backend_vk_init(device); |
There was a problem hiding this comment.
you are initializing EVERY vk device here (and leak it)
There was a problem hiding this comment.
I copied this from the original code verbatim in order to ensure that behavior prior to my changes was preserved. I just fixed it though. Instead of iterating over all device indexes, initializing all of them, and only keeping the last one, it instead simply picks the last index and initializes the corresponding backend device.
stable-diffusion.h
Outdated
| bool keep_vae_on_cpu, | ||
| int model_backend_index = -1, | ||
| int clip_backend_index = -1, | ||
| int vae_backend_index = -1); |
There was a problem hiding this comment.
C can not have default params
Removed whitespace, changed the default vulkan device initialization to use the last device without initializing all other previous devices.
|
Everything should be fixed now |
Green-Sky
suggested changes
Nov 20, 2024
examples/cli/main.cpp
Outdated
| @@ -5,6 +5,7 @@ | |||
| #include <random> | |||
| #include <string> | |||
| #include <vector> | |||
| #include <stdexcept> | |||
There was a problem hiding this comment.
There are no exceptions used.
There was a problem hiding this comment.
Anything else that needs changing?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The ability to specify separate backend device indexes for CLIP, Unet, and VAE was implemented. This works for all backends that accept a device index as an initialization parameter (CUDA, Vulkan, and SYCL.) The aim of this was to support splitting the parts of the model across multiple GPUs.