ROCm Testing

[bdashore3]

ROCm is supported on tabbyAPI, but there's no real way to test start scripts. Figure out how to detect if an AMD GPU is present on a system to install the appropriate requirements.

[Essoje]

At least on Linux, you'll want to check if /opt/rocm/bin/rocminfo can be run, and if it does not exist, which rocminfo should give you an indicator that ROCM is installed somewhere. Both need to be checked in this order, as there are a few non-standard installations for ROCM that don't use /opt, or rocminfo itself might not be in $PATH.
If one or the other exists, then we call it to get the architecture of the card by using /opt/rocm/bin/rocminfo | grep -m1 gfx | awk -F':' '{print $2}' | sed 's/ //g'. In my case, it returns gfx1030 for my AMD Radeon RX 6900 XT.
That should be enough to confirm the existence of an AMD graphics card.

[bdashore3]

Thanks for the advice!

I'd use rocminfo to check, but it seems like torch bundles rocm with it (according to @Baysul) which means that the rocminfo utility can't be found on some systems.

[Essoje]

cat /proc/bus/pci/devices | grep -o amdgpu returning amdgpu instead of an empty string is a sure-fire way to know if a system has access to an AMD GPU. It's the more bare-bones way to do it and cat + grep are unlikely to be missing from a linux installation, but it doesn't confirm it's going to work, just that it's there.
Edit: fixed the bash command.

[cikkle]

When I try that command on some of my systems it seems to pick up my integrated graphics. I remember last year when I started playing with llama, I had to fight with and edit ooba's startup script because it would see my 7950X3D CPU, install rocm, and crash trying to load the model into limited shared memory while ignoring my nvidia gpu.

If it's an option, I don't think it would be bad to simply ask the user interactively during the script whether they're using nvidia/amd.

[bdashore3]

While that's a good point, the start script applies to both new users and existing users alike. Asking the user each time if nvidia or AMD is being used can be cumbersome. However, it looks like this is the only certain way to write a start script that can get ROCm or CUDA right 100% of the time.

[bdashore3]

Fixed in #88. There's no real platform agnostic way to detect the GPU and on top of that, pytorch can install runtimes by itself as well. Fallback to asking the user inside the start script and saving the preferences.